Systems and methods for using multiple MEMS projectors to determine a position of a photosensor of an HMD or another controller

ABSTRACT

A method for determining an orientation of a photosensor of a controller with respect to a projector is described. The method includes generating, by a beam generator of the projector, a beam. The method further includes modifying a direction of travel of the beam using a micro-electro-mechanical systems (MEMS) mirror that moves in a pattern, detecting the beam, calculating a time at which the beam is detected, and determining based on the pattern and the time an orientation of the beam to determine the orientation of the photosensor.

CLAIM OF PRIORITY

This application claims priority, under 35 U.S.C. § 119(e), from U.S.Provisional Patent Application No. 62/202,766, filed on Aug. 7, 2015,and entitled “Systems And Methods For Using MEMS Projectors To DeterminePosition Of A Controller”, which is incorporated herein by reference inits entirety for all purposes.

FIELD

The present disclosure relates to systems and methods for using multipleMicro-Electro-Mechanical Systems (MEMS) mirror projectors to determine aposition of a photosensor of a controller.

BACKGROUND

The video gaming industry is gaining traction. With the advancement oftime, more people are playing video games using gadgets, e.g.,controllers, cell phones, head-mounted displays, etc. To play the videogames, a position of a user is determined.

It is within this context that embodiments described in the presentdisclosure arise.

SUMMARY

In one embodiment, a system including a Micro-Electro-Mechanical Systems(MEMS) mirror is used to trace a beam across a measurement space. Manypatterns of beam steering are possible using the MEMS mirror. However,as an example, a Lissajous variant pattern is applied. The Lissajousvariant pattern is to drive the MEMS mirror in a resonant mode ofoperation, e.g., move the MEMS mirror along a y axis of the MEMS mirrorbelow a resonant frequency of the MEMS mirror and to move the MEMSmirror along an x axis of the MEMS mirror above the resonant frequency.Criteria for repetition of the Lissajous variant pattern are set. Whenthe MEMS mirror is a rotating mirror, measurements, e.g., pulses of adetected beam hit signal, etc., are limited to a pre-determinedfrequency, e.g., 60 hertz, a range between 55 hertz and 65 hertz, etc.,of the Lissajous variant pattern.

By using the beam, an entire measurement space, e.g., volume, etc., isswept with one or two rotating MEMS mirrors at this rate. The beam movesmore quickly than a line to scan the entire measurement space. Bydriving the MEMS mirror close to its resonant frequency, the beam isused and desired update rates are achieved for tracking the beam. TheMEMS mirror moves in the resonant mode as long as frequencies ofmovement of the MEMS mirror along the x and y axes are within a fewpercentage of a resonant peak. A slight difference in the frequencies ofmovement of the MEMS mirror along the x and y axes in an order of, e.g.,1/180^(th), 1/200^(th), 1/170^(th), etc., generates the Lissajousvariant pattern that fills the entire measurement space.

By using the beam instead of the line, a synchronized timing of thesystem is used to determine an orientation of the beam. The systemfurther includes a receiver, e.g., a head-mounted display, a hand-heldcontroller, etc., which includes one or more photosensors and the systemincludes a projector having the MEMS mirror to determine the orientationof the beam. Clock signals of both the projector and the receiver aresynchronized to determine the beam orientation. In an embodiment,multiple photosensors in a rigid topology or multiple projectors with aknown baseline facilitates a method of triangulation orPerspective-n-Point to determine a three-dimensional (3D) pose of thephotosensor(s) with respect to the projector(s).

The Lissajous variant pattern is executed to facilitate a derivation ofthe orientation of the beam from a timing of the photosensor detectedbeam hit signal. With a configuration of a width of the beam and thefrequencies, of the MEMS mirror, associated with the Lissajous pattern,the photo sensor detects intensity peaks of the beam several times permeasurement period, e.g., a frequency of the Lissajous pattern repeat,etc. The timings of the intensity peaks allow for good overall averagingof the orientation of the beam. So even if there is noise in precisionof the timings of the intensity peaks, by having a pre-determinednumber, e.g. 3 to 5, etc. of the intensity peaks per measurement periodof the frequency of the Lissajous pattern, the noise is reduced toobtain the orientation of the beam.

In one embodiment, the projector generates infra-red light or visiblelight as the beam.

In an embodiment, multiple projectors are configured to perform timedivision multiplexing. For example, two projectors have alternateemission of beams so that one of the projectors does a sweep of theLissajous variant pattern and then another one of the projectorsperforms a sweep of the Lissajous variant pattern.

In one embodiment, the time division multiplexing is performed at afiner grain, e.g., a period of the resonance mode of operation, etc.

In an embodiment, to achieve the synchronization, a global strobeillumination is used during each sweep of the Lissajous pattern orduring fractions of the sweep. In one embodiment, to achieve thesynchronization, wired communication or radio frequency (RF)communication is used.

In one embodiment, a beam is modulated by embedding a repeating bit codeinto the beam. In this way, the photo sensor detects several bits of therepeating bit code to differentiate between multiple uniquely codedprojectors.

In an embodiment, a correlation to determine the repeating bit code isused to improve the time synchronization and consequently improve themeasurement of the beam orientation when detected by the photosensor.

As an alternative to time division multiplexing, frequency multiplexingis used. For example, two different light frequencies are used and thenfilters are placed on photosensors so that one of the photosensorsdetects one of the frequencies from a first one of the projectors butnot a second one of the projectors and another one of the photosensorsdetects another one of the frequencies from the second projector but notthe first projector. Both the first and second projectors are onsimultaneously.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a diagram of an embodiment of a system to illustratedetermination of a beam orientation of a beam to further determine aposition of a head-mounted display (HMD) with respect to a projector.

FIG. 1B is a diagram of an embodiment to illustrate frequencies ofmovement of a Micro-Electro-Mechanical Systems (MEMS) mirror.

FIG. 2 is a diagram of an embodiment of a system to illustrate thatinstead of the HMD, a hand-held controller is scanned by the beamemitted by the projector to determine the beam direction to furtherdetermine a pose of the hand-held controller with respect to theprojector.

FIG. 3 is a block diagram to illustrate a computing device including aprocessor that executes one or more of the operations described herein.

FIG. 4A is a flowchart of an embodiment of a method for determining thebeam orientation.

FIG. 4B is a flowchart of an embodiment of a method for determining thebeam orientation and the pose of the HMD.

FIG. 4C is a continuation of the flowchart of FIG. 4B.

FIG. 4D is a diagram of an embodiment of a system to illustrate adetermination of the beam orientation.

FIG. 4E is a diagram of an embodiment of a system to illustrate thateach time at which the beam is generated corresponds to a differentorientation of the beam with respect to an xyz co-ordinate system of theprojector.

FIG. 5 is a diagram of an embodiment of a Lissajous pattern.

FIG. 6A is a diagram of an embodiment to illustrate a measured locationof a photosensor with respect to an actual location of the photosensorto determine a location of a beam hit within a diameter of the beam.

FIG. 6B is a diagram of an embodiment to illustrate that the location ofthe beam hit is determined using an intersection of a diameter of aportion of the Lissajous pattern and a diameter of another portion ofthe Lissajous pattern when there is no appreciable movement in thephotosensor between detections.

FIG. 6C is a diagram of an embodiment to illustrate a linear correctionin a location of the beam hit determined using an intersection of adiameter of a portion of the Lissajous pattern and a diameter of anotherportion of the Lissajous pattern when there is movement in thephotosensor.

FIG. 6D shows a more generalized correction based around the formula ofthe Lissajous curves and the correction is more accurate than a simplelinear correction.

FIG. 7A is a diagram of an embodiment of a system for illustrating amethod of triangulation to determine a position of the photosensor withrespect to the projectors and for illustrating time divisionmultiplexing between multiple projectors to identify one of theprojectors from which the beam is detected by the photosensor of theHMD.

FIG. 7B is a diagram of an embodiment of a system for illustratinganother method to determine a pose of a rigid topology of photosensorswith respect to a single projector using a method, such as,Perspective-n-Point.

FIG. 8A is a diagram of an embodiment of a system to illustrate amodulation of beams emitted by the projectors to identify theprojectors.

FIG. 8B is a diagram of an embodiment of a modulated pulse of a detectedbeam hit signal that is generated by the photosensor upon detecting thebeam hit from the beam as the beam moves across the photosensor.

FIG. 8C is a diagram to illustrate than a Maximum Length Sequence (MLS)bit sequence for the projector repeats many times along the Lissajouspattern.

FIG. 8D is a diagram of an embodiment of a correlation filter that isapplied by the processor of the computing device to detect a portion ofthe MLS bit sequence.

FIG. 8E is a diagram of an embodiment illustrating an application of athreshold to the correlation filter used to determine the match of aportion of the MLS bit sequence.

FIG. 8F is a diagram of an embodiment of a portion of the Lissajouspattern to illustrate a number of times a match between the correlationfilter and the MLS bit sequence occurs along the Lissajous pattern.

FIG. 8G is a diagram of an embodiment of a portion of the Lissajouspattern to illustrate an MLS-sequence technique in which the length ofthe MLS bit sequence detected is used to determine an improved locationof the photosensor within the diameter of the Lissajous pattern.

FIG. 9 is a diagram of an embodiment of a system to illustrate a methodof synchronization of clock signals between a device and another device.

FIG. 10 is a flowchart of an embodiment of a method for synchronizingclock signals of the projector and the HMD by extracting a frequency ofa modulating signal from the detected beam hit signal that is generatedby the photosensor.

FIG. 11 is a diagram to illustrate a method for averaging of intensitiesof detected beam hit signals to determine the orientation of the beam.

FIG. 12 is a flowchart of an embodiment of a method for synchronizationof the clock signal that is generated within the HMD with the clocksignal that is generated within the projector.

FIG. 13 an isometric view of an embodiment of the HMD.

FIG. 14 is a diagram of an embodiment of a system to illustrate aninteraction of a user with a virtual environment by using the HMD andthe hand-held controller.

FIG. 15 is an isometric view of an embodiment of the HMD.

FIG. 16 is a diagram used to illustrate access of a virtual environmentvia a computer network.

FIG. 17 illustrates the user wearing the HMD to access the virtualenvironment.

FIG. 18 is a diagram to illustrate example components of the HMD.

FIG. 19 illustrates an embodiment of an Information Service Providerarchitecture.

FIG. 20 is a diagram of an embodiment of the HMD having multiplephotosensors.

DETAILED DESCRIPTION OF THE EMBODIMENTS

FIG. 1A is a diagram of an embodiment of a system 100 to illustratedetermination of a beam direction θ to further determine a position of ahead-mounted display (HMD) 102 with respect to a projector 104. Thesystem 100 includes the projector 104, the HMD 102, a computing device107, and a hand-held controller 115. The HMD 102 is worn on head of auser 106 and the projector 104 is located within an environment, e.g., aroom, a location, etc., in which the user 106 is located.

The computing device 107 is coupled to the HMD 102 via a link 109. Anexample of a link, as used herein, includes a wired link, e.g., a cable,etc. Another example of a link includes a wireless link. For example, awireless communication device located within the computing device 107applies a wireless communication protocol, e.g., Wi-Fi, Bluetooth, radiofrequency (RF) protocol, etc., to communicate via the wireless link witha wireless communication link located within the HMD 102.

Similarly, the computing device 107 is coupled to the projector 104 viaa link 111. For example, the wireless communication device locatedwithin the computing device 107 applies the wireless communicationprotocol to communicate via the wireless link with a wirelesscommunication device located within the projector 104. Furthermore, thecomputing device 107 is coupled to the hand-held controller 115 via alink 113. For example, the wireless communication device located withinthe computing device 107 applies the wireless communication protocol tocommunicate via the wireless link with a wireless communication linkdevice within the hand-held controller 115.

In one embodiment, a wireless communication device, as used herein, is aprocessor.

A beam generator 108 of the projector 104 emits a beam 110, which isreflected by a Micro-Electro-Mechanical Systems (MEMS) mirror 112towards the HMD 102. The MEMS mirror 112 is driven by a driver 114,e.g., an actuator, a magnetic actuator, a thermal actuator, a magnet,etc., at one or more frequencies. For example, the MEMS mirror 112 isdriven to move, e.g., translate, rotate, swing, etc., with respect to anX-axis of the MEMS mirror 112 and/or with respect to a Y-axis of theMEMS mirror 112. It should be noted that in some embodiments, axes,e.g., X-axis, Y-axis, etc., of the MEMS mirror 112 are different fromaxes, e.g., x-axis, y-axis, etc., of a projector. Examples of the beamgenerator 108 include a laser beam light source, an invisible lightsource, a visible light source, etc. The MEMS mirror 108 is a dual-axismirror, e.g., a microscanner, etc. In one embodiment, the driver 114 isconnected to a processor of the projector 104 and the processor of theprojector 104 provides a control signal to the driver 114 to drive theMEMS mirror 112 at a repetition frequency, e.g., a frequency at which apattern, e.g., a Lissajous pattern, a raster scan pattern, etc. isrepeated by the MEMS mirror 112. Upon receiving the control signal, thedriver 114 generates a field, e.g., a magnetic field, an electric field,etc., to control movement of the MEMS mirror 112.

The movement of the MEMS mirror 112 reflects the beam 110 at an angle θxwith respect to an x-axis of the projector 104, an angle θy to a y-axisof the projector 104, and an angle θz relative to a z-axis of theprojector 104. In some embodiments, the z-axis of the projector 104 isdefined by a direction of the beam 110 after being reflected from theMEMS mirror 112 when there is no displacement of the MEMS mirror 112.The angles θx, θy, and θz provide an orientation of the beam 110 withrespect to an origin of an xyz co-ordinate system of the projector 104.The origin of the xyz co-ordinate system is at an intersection of thex-axis of the projector 104, the y-axis of the projector 104, and thez-axis of the projector 104. A y-axis is perpendicular to an x-axis andboth the x and y axes are perpendicular to a z-axis. The orientation ofthe beam 110 when detected by the photosensor 105 a provides anorientation of the photosensor 105 a with respect to the xyz co-ordinatesystem of the projector 104.

Multiple photosensors 105 a, 105 b, and 105 c, e.g., photodiodes, etc.,located on, e.g., fitted to, placed on, affixed to, etc., an outersurface of the HMD 102 detect the beam 110 to generate electrical sensorsignals, which are then used by the computing device 107 to determine apose of the HMD 102 from the projector 104, e.g., a distance along thebeam 110 from the origin of the xyz co-ordinate system of the projector104, etc., and an orientation of the HMD 102 with respect to the xyzco-ordinate system.

In one embodiment, the HMD 102 includes any other number of photosensorson its outer surface for sensing the beam 110. It should be noted thatthe outer surface of the HMD 102 is located opposite to an inner surfaceof the HMD 102. The inner surface of the HMD 102 faces eyes of the user106 when the HMD 102 is worn by the user 102.

In an embodiment, instead of the Lissajous pattern, the raster scan isused.

In one embodiment, the beam direction, e.g., the angles θx, θy, and θz,etc., is measured with respect to a horizontal axis, e.g., the x-axis ofthe projector 104, etc., the vertical y-axis of the projector 104, andthe forward z-axis of the projector 104.

In an embodiment, a beam generated by a projector, described herein, isa visible light beam, an infrared light beam, or a beam of another lightfrequency.

In one embodiment, an orientation of the beam hitting photosensor 105 awith respect to the projector 104 is represented by the angles θx, θy,and θz formed by the beam 110 with respect to the corresponding x-axis,y-axis, and z-axis of the projector 104 when the beam 110 is detected bythe photosensor 105 a.

In an embodiment, a position of the HMD 102 with respect to theprojector 104 is the same as a position of the HMD 102 with respect tothe origin of the xyz co-ordinate system. Moreover, an orientation ofthe HMD 102 with respect to the projector 104 is the same as theorientation of the HMD 102 with respect to the xyz co-ordinate system.

In one embodiment, a wide field of view is obtained by sweeping the beam110. For example, when the MEMS mirror 112 does not have a sufficientpre-determined amount of deflection, a wide angle lens is used betweenthe MEMS mirror 112 and an output of the projector 104 to output thebeam 110 at a wide angle from the projector 104.

In an embodiment, an effect of the wide angle lens on beam divergence isdetermined.

In one embodiment, the beam 110 is modeled as a Gaussian beam.

In an embodiment, a part of the Lissajous pattern that is sparsecompared to other parts of the Lissajous pattern is along the z axis ofthe projector 104.

In one embodiment, the beam 110 fills a space to avoid gaps. The spaceis a portion of the environment. For example, the space is an amount ofthe environment covered by movement of the beam 110 in the environment.As another example, the space is an amount of the environment in which amovement of the HMD 102 is detected by a sweep of the beam 110. Anamount of the space filled by the beam 110 depends on a diameter of thebeam 110, a divergence of the beam 110 from the projector 104, and acollection area of the photosensor 105 a. The collection area of thephotosensor 105 a detects the beam 110. Increasing a diameter of thebeam 110 allows for greater coverage but at a cost of measurementprecision and power requirements of generating the beam 110.

In one embodiment, instead of or in addition to the diameter of the beam110, a diameter of the photosensor 105 a is increased.

In an embodiment, the photosensor 105 a is at an oblique angle to thebeam 110 or even partially occluded by a design of the HMD 112 so thatthe photosensor 105 a cannot detect the beam 110. Without any lensingcap on the photosensor 105 a, a flat disc of light of the beam 110becomes an ellipse as it falls on the photosensor 105 a as it turns awayfrom the projector 104. The lensing cap improves an angular sensitivityof the photosensor 105 a.

In an embodiment, the divergence of the beam 110 matches an opticalprojection frustum divergence so that at any particular distance fromthe projector 104, the Lissajous pattern has the same percentages offilled space. However, in one embodiment, in which the match does notoccur, e.g., there is overlap at 1 meter range and no overlap at 3 meterrange from the projector 104, etc., and the discrepancy in overlap isaccounted for.

In one embodiment, the HMD 102 is coupled to the projector 104 via alink 117.

In some embodiments, the terms pose and position are usedinterchangeably herein.

FIG. 1B is a diagram of an embodiment of the MEMS mirror 112 toillustrate various frequencies of movement of the MEMS mirror 112. TheMEMS mirror 112 is driven with respect to, e.g., about, etc., the X-axisof the MEMS mirror 112 at a frequency f1 and driven with respect to theY-axis of the MEMS mirror 112 at a frequency f2. In an embodiment, thefrequency f2 is different from the frequency f1. For example, when f1 is35060 hertz (Hz), f2 is 35000 Hz. As another example, when f1 is 34,620Hz, f2 is 34,560 Hz. As yet another example, when f1 is at a resonantfrequency of the MEMS mirror 112, f2 is slower than the resonantfrequency. As another example, a resonant frequency of the MEMS mirror112 is an average of f1 and f2.

When f1 is less than f2 by a substantial pre-determined amount, anapproximate raster scan of the HMD 102 is performed by the beam 110.When f1 is substantially the same as f2, then a more symmetricalLissajous scan pattern emerges.

In one embodiment, the MEMS mirror 112 is driven in several modes. Forexample, to quickly scan with a large angular displacement in the MEMSmirror 112, one mode is a resonance mode in which driving frequencies atwhich the MEMS mirror 112 is driven by the driver 114 are close to anatural mechanical resonance of the MEMS mirror 112. Accordingly, theLissajous pattern is efficient in terms of overall coverage of the spacein a certain time frame.

FIG. 2 is a diagram of an embodiment of a system 200 to illustrate thatinstead of or in addition to the HMD 102, a hand-held controller 202 isscanned by the beam 110 of the projector 104 to determine the beamorientation, which includes the angles θx, θy, and θz, to furtherdetermine a pose of the hand-held controller 202 with respect to theprojector 104. Examples of a hand-held controller, as used herein,include a game controller, a game pad, a paddle, a joystick, atrackball, a gun-shaped controller, a bow-shaped controller, a steeringwheel, a yoke, a keyboard, a mouse, a tablet, a cell phone, etc. In oneembodiment, the HMD 102 is an example of a controller.

The hand-held controller 202 has multiple photosensors 204 a and 204 b,e.g., photodiodes, etc., on its surface for sensing the beam 110. Insome embodiments, the hand-held controller 202 includes any other numberof photosensors, e.g., three, four, five, ten, twenty, etc., on itssurface for sensing the beam 110. The projector 104 operates in asimilar manner to that described above to generate the beam 110 and todirect the beam 110 at the angles θx, θy, and θz with respect to theprojector 104. The photosensor 204 a on the hand-held controller 202senses the beam 110 to generate the electrical sensor signals, which arethen used to determine an orientation including the angles θx, θy, andθz of the beam 110 with respect to the projector 104 when thephotosensor 204 a senses the beam 110. The orientation of the beam 110with respect to the projector 104 when the beam 110 is sensed by thephotosensor 204 a provides an orientation of the photosensor 204 a withrespect to the origin of the xyz co-ordinate system of the projector104. The orientation is then used to determine a position of thephotosensor 204 a from the origin of the xyz co-ordinate system of theprojector 104. In this manner, orientations of the photosensors 204 aand 204 b are determined with respect to the xyz co-ordinate system ofthe projector 104 to determine a pose of the hand-held controller 202with respect to the xyz co-ordinate system of the projector 104. Thepose of the hand-held controller is calculated when at least threephotosensors on the handheld controller have known directions withrespect to the projector's coordinate system using a technique, such as,for example, Perspective-n-Point, etc. To illustrate, a square ofdistance between the photosensors 105 a and 105 b is expressed as afirst relation expressed as 1² _(ab)=a²+b²−2ab(cos θ_(ab)), where a is adistance of the photosensor 105 a from the origin of the xyz co-ordinatesystem of the projector 104, b is a distance of the photosensor 105 bfrom the origin of the xyz co-ordinate system of the projector 104, cosis a cosine function, and θ_(ab) is an angle between the distances a andb. Moreover, a square of distance between the photosensors 105 b and 105c is expressed as a second relation expressed as 1² _(bc)=b²+c²−2bc(cosθ_(bc)), where c is a distance of the photosensor 105 c from the originof the xyz co-ordinate system of the projector 104, and θ_(bc) is anangle between the distances b and c. Also, a square of distance betweenthe photosensors 105 a and 105 c is expressed as a third relationexpressed as 1² _(ac)=a²+c²−2ab(cos θ_(ac)), where θ_(ac) is an anglebetween the distances a and c. The distances lab, lbc, and lac arepre-determined by the processor 302. Moreover, the angles θ_(ac),θ_(bc), and θ_(ac) are determined by the processor 302 from theorientations of the photosensors 105 a, 105 b, and 105 c with respect tothe origin of the xyz co-ordinate system of the projector 104. Theprocessor 302 solves for the distances a, b, and c by solving the threerelationships.

In an embodiment, a position of the hand-held controller 202 withrespect to the projector 104 is the same as a position of the hand-heldcontroller 202 with respect to the origin of the xyz co-ordinate systemof the projector 104. Moreover, an orientation of the hand-heldcontroller 202 with respect to the projector 104 is the same as theorientation of the hand-held controller 202 with respect to the xyzco-ordinate system of the projector 104.

In one embodiment, the hand-held controller 202 is the same as thehand-held controller 115 (FIG. 1A) except that the hand-held controller202 includes multiple photosensors 204 a, 204 b and more.

FIG. 3 is a block diagram to illustrate a computing device 107 thatperforms one or more operations, e.g., mathematical operations,calculation operations, determination operations, etc., describedherein. The computing device 107 includes a processor 302, a memorydevice 304, a network interface controller 306, and multipleinput/output (I/O) ports 308 a, 308 b, 308 c, and 308 d. An example ofthe network interface controller 306 includes a network interface card(NIC) for accessing a computer network, e.g., a local area network(LAN), a wide area network (WAN) such as the Internet, or a combinationthereof, etc. An example of each the I/O ports 308 a-308 d include aserial port, a parallel port, or a Universal Serial Bus (USB) port. Theserial port transfers data in a serial manner, e.g., one bit at a time,etc., between the processor 302 and an I/O device, e.g., the HMD 102,the hand-held controller 115 (FIG. 1A), the projector 104 (FIG. 1A),etc., that is coupled to the serial port. The parallel port transfersdata in a parallel manner, e.g., multiple bits at a time, etc., betweenthe processor 302 and the I/O device that is coupled to the parallelport. The USB port applies the USB protocol to transfer data between theprocessor 302 and the I/O device that is coupled to the USB port. Aprocessor, as used herein, refers to a central processing unit (CPU), anapplication specific integrated circuit (ASIC), or a programmable logicdevice (PLD) is used, and these terms are used interchangeably herein.Examples of a memory device, as used herein, include a read-only memory(ROM), a random access memory (RAM), a hard disk, a volatile memory, anon-volatile memory, a redundant array of storage disks, a Flash memory,etc. Examples of a computing device include a game console, a desktopcomputer, a laptop computer, a smart television, a tablet, a cell phone,etc.

In one embodiment, the computing device 107 is accessible via thecomputer network by the HMD 102 and/or the projector 104. For example,the computing 300 device is a portion of a cloud network or a serverrack.

In an embodiment, any of the processing operations described herein asbeing performed by the computing device 107 are performed in the cloudnetwork.

FIG. 4A is a flowchart of an embodiment of a method 400 for determiningthe orientation of the photosensor 105 a with respect to the origin ofthe xyz co-ordinate system. The photosensor 105 a located on the HMD 102generates, in an operation 402, a detected beam hit signal, e.g., asignal having a pulse having a light beam intensity peak, a signalhaving a pulse having a light beam intensity local maxima, an electricalsensor signal, a signal having a pulse having a maximum amplitude, etc.,when the beam 110 is sensed by the photosensor 105 a. The photosensor105 a sends the detected beam hit signal to the processor 302 of thecomputing device 107. The processor 302 determines, in an operation 404,the orientation of the photosensor 105 a with respect to the origin ofthe xyz co-ordinate system based on the detected beam hit signalaccording to a method 450 described below with respect to FIG. 4B.

FIG. 4B is a flowchart of an embodiment of the method 450 fordetermining the orientation of the photosensor 105 a with respect to theorigin of the xyz co-ordinate system of the projector 104. It should benoted that during the projection and scanning of beam 110 from theprojector 104 (FIG. 1A), a clock signal generated by a clock source,e.g., a crystal oscillator, a clock oscillator, a clock signalgenerator, etc., of the projector 104 is continuously synchronized, inan operation 452, with a clock signal generated by a clock source, e.g.,a crystal oscillator, a clock oscillator, a clock signal generator,etc., of the HMD 102. For example, the clock signal generated by theclock source of the HMD 102 is within a pre-determined phase differencefrom the clock signal generated by the clock source of the projector104. As another example, the clock signal of the HMD 102 has the samephase as that of the clock signal of the projector 104. A location ofmovement of the MEMS mirror 112 and the beam 110 in a pattern, e.g., theLissajous pattern, the raster scan pattern, etc., is determined usingthe clock signal generated within the projector 104 and a time ofdetection of the beam 110, e.g., a time of peak intensity of the pulseof the beam 110, a time of maximum amplitude of the pulse, a time ofamplitude peak of the pulse, etc., by the photosensor 105 a. The time ofdetection of the beam 110 is measured by the clock signal generatedwithin the HMD 102.

The synchronization of the clock signal generated by the projector 104with the clock signal generated by the HMD 102 is performed by theprocessor 302 of the computing device 107. For example, the processor302 of the computing device 107 communicates via the link 111 (FIG. 1A)with the processor of the projector 104 to receive a phase of the clocksignal of the projector 104. The processor of the projector 104 iscoupled to the clock source of the projector 104. Moreover, theprocessor 302 of the computing device 107 communicates via the link 109(FIG. 1A) with a processor of the HMD 102 to provide the phase of theclock signal to the processor of the HMD 102. Upon receiving the phase,the processor of the HMD 102 sends a signal to the clock source of theHMD 102 to modify a phase of the clock signal of the HMD 102 so that thephases of the clock signal of the projector 104 and of the clock signalof the HMD 102 are synchronized.

In one embodiment, the processor of the projector 104 sends a phase ofthe clock signal generated by the clock source of the projector 104 viathe link 117, e.g., a radio frequency signal link, a wireless link, awired communication medium, etc., between the projector 104 and the HMD102 to the processor of the HMD 102. The processor of the HMD 102receives the phase of the clock signal of the projector 104 and modifiesa phase of the clock signal generated by the clock source of the HMD 102to match with the phase of the clock signal of the projector 104.

In an embodiment, the processor of the HMD 102 sends a phase of theclock signal generated by the clock source of the HMD 102 via the link117 between the projector 104 and the HMD 102 to the processor of theprojector 104. The processor of the projector 104 receives the phase ofthe clock signal of the HMD 102 and modifies a phase of the clock signalgenerated by the clock source of the projector 104 to match with thephase of the clock signal of the HMD 102.

In one embodiment, when the clock signals of the HMD 102 and theprojector 104 are synchronized, there is less than a pre-determinedphase difference between phases of the clock signals.

In an embodiment, a processor of a device, e.g., the HMD 102, theprojector 104, etc., sends a phase, of a clock signal, received fromanother device, e.g., the projector 104, the HMD 102, etc., to aphase-locked loop (PLL) of the device and the PLL matches the phase to aphase of a clock signal generated by the device.

The photosensor 105 a sends the detected beam hit signal to theprocessor of the HMD 102. The processor of the HMD 102 converts thedetected beam hit signal from an analog form to a digital form and sendsthe digital form of the beam hit signal timing via the link 109 (FIG.1A) to the computing device 107. Moreover, the processor of the HMD 102obtains a time of hit of the beam 110, e.g., a time of occurrence of thepeak magnitude of the pulse of the beam 110, a time of occurrence of thelocal maxima of intensity of the pulse of the beam 110, a time of anincrease in intensity of the beam 110 beyond a pre-determined limit, atime of generation of the pulse of the detected beam hit signal, etc.,on the photosensor 105 a from the clock source of the HMD 102 and sendsthe time of hit via the link 109 to the processor 302 of the computingdevice 107. The processor 302 of the computing device 107 receives, inan operation 454, the detected beam hit signal and the time of hit ofthe beam 110 on the photosensor 105 a. The time of hit of the beam 110is a time at which the beam 110 is detected by the photosensor 105 a andis measured by the clock source of the HMD 102.

The processor 302 determines, in an operation 456, a location of thebeam 110 within the pattern based on the time of hit and a time of startof the pattern. In one embodiment, the time of start of the pattern isobtained by the processor 302 from the processor of the projector 104via the link 111 (FIG. 1A), and is a time at which the MEMS mirror 112moves to create or repeat the pattern. The processor of the projector104 obtains the time of start of the pattern from the clock source ofthe projector 104. The clock source of the projector 104 measures thetime of start of the pattern. As an example of the determination of thelocation of the beam 110 within the pattern, the processor 302 of thecomputing device 107 determines from the time of the start of thepattern, the frequency f1, the frequency f2, and the time of hit, thelocation of the beam 110 within the pattern. To illustrate, theprocessor 302 of the computing device 107 determines that the time ofstart corresponding to an initial location of the beam 110 in thepattern. Moreover, the processor 302 determines that the initiallocation changes according to the frequencies f1 and f2 of the MEMSmirror 112 to achieve the location of the beam on the pattern 110 at thetime of hit. In one embodiment, a correspondence, e.g., a one-to-onerelation, etc., among the frequencies f1 and f2, the time of hit, thetime of start, and the location of the beam 110 in the pattern is storedin the memory device 304 of the computing device 107. The correspondenceis accessed by the processor 302 to determine the location of the beam110 in the pattern from the time of start, the frequencies f1 and f2,and the time of hit.

FIG. 4C is a continuation of the flowchart of FIG. 4B and of the method450 of FIG. 4B. In an operation 458, the processor 302 of the computingdevice 107 determines the orientation of the photosensor 105 a withrespect to the origin of the xyz co-ordinate system of the projector 104from the location on the pattern. For example, as shown in FIG. 4D, theclock signals of the projector 104 and an HMD 470 are synchronized. TheHMD 470 is an example of the HMD 102 (FIG. 1A). The projector 104 emitsthe beam 110 at times T1 through T8 as the beam 110 follows the pattern.The times T1 through T8 are measured by the clock source of theprojector 104. Each time T1 through T8 corresponds to a differentorientation of the beam 110 with respect to the xyz co-ordinate systemof the projector 104. It should be noted that the generation of the beam110 is shown along the y-axis of the projector 104 for simplicitypurposes. The beam 110 scans to form the pattern along the x-axis of theprojector 104 and also along the y-axis of the projector 104 as shown bythe xyz co-ordinate system in FIG. 4D. The photosensor 105 a detects thebeam 110 at the time T3. As illustrated with reference to FIG. 4E, basedon a one-to-one correspondence, e.g., mapping, relationship, link, etc.,with the time T3, the processor 302 of the computing device 107determines that the orientation of the beam 110 is at angles θx1, θy1,and θz1 with respect to the origin of the xyz co-ordinate system of theprojector 104. For each repetition of the pattern, there is a one-to-onecorrespondence between the time of hit, the angles θx, θy, and θz, andthe location of the beam 110 in the pattern. The user 106 then moves tomove the HMD 102 with respect to the origin of the xyz co-ordinatesystem of the projector 104. After the movement of the HMD 102, thephotosensor 105 a detects the beam 110 at the time T4. From a one-to-onecorrespondence with the time T4, the processor 302 determines that theorientation of the beam 110 is at angles θx2, θy2, and θz2 with respectto the origin of the xyz co-ordinate system of the projector 104.

As shown in FIG. 4E, each of the photosensors of an HMD 470 is angledwith respect to a front face of the HMD 470, which is an example of theHMD 102. For example, the photosensor 105 a is angled to form an angleϕ1 with respect to a y-axis of the HMD 470. A direction of the y-axis ofthe HMD 470 changes with a movement of the HMD 470.

Referring back to FIG. 4C, the processor 302 further determines, in anoperation 460, that the photosensor 105 a, which detects the beam 110,is located somewhere along the beam 110. The processor 302 applies, inan operation 462, a method of triangulation, described below, todetermine a position of the photosensor 105 a with respect to the xyzco-ordinate system of the projector 104 (FIG. 1A).

In some embodiments an orientation of a photosensor that is not angledwith respect to, e.g., is parallel to, etc., the front face of the HMD470 detects the beam 110 is the same as that of the beam 110. In theseembodiments, the orientation of the photosensor and the beam 110 is withrespect the xyz co-ordinate system of the projector 104. In variousembodiments, an orientation of a photosensor that is angled with respectto the front face of the HMD 470 is a combination of an angle formed bythe photosensor with respect to the front face of the HMD 470 and theangle formed by the beam 110 with respect to the xyz co-ordinate systemof the projector 104.

FIG. 4D is a diagram of an embodiment of a system to illustrate thedetermination of the orientation of the photosensor 105 a with respectto the projector 104. The system illustrated in FIG. 4D includes the HMD470 and the projector 104. The projector 104 emits the beam 110 at thetimes T1 through T8 during a scan to form the pattern and the beam 110is detected by one or more photosensors of the HMD 470.

FIG. 4E is a diagram of an embodiment of a system to illustrate thateach of the times T1 through T8 at which the beam 110 is generatedcorresponds to a different orientation of the beam 110 with respect tothe xyz co-ordinate system of the projector 104. For example, the timeT3 corresponds to a first combination of values of the angles θx, θy,and θz, and the time T4 corresponds to a second combination of values ofthe angles θx, θy, and θz. It should be noted that at least one of thevalues of the first combination is different from at least one of thecorresponding values of the second combination. For example, the valueof the angle θx of the first combination is different from the value ofthe angle θx of the second combination, and/or the value of the angle θyof the first combination is different from the value of the angle θy ofthe second combination, and/or the value of the angle θz of the firstcombination is different from the value of the angle θz of the secondcombination.

FIG. 5 is a diagram of an embodiment of a Lissajous pattern 500. TheLissajous pattern 500 has a central region 502 and an edge region 504.As shown, angles between traversal of the Lissajous pattern 500 in thecentral region 502 are greater than angles between traversal of theLissajous pattern 500 in the edge region 504. For example, an anglebetween a portion 506 a of the Lissajous pattern 500 and a portion 506 bof the Lissajous pattern 500 is greater than an angle between theportion 506 b and a portion 506 c of the Lissajous pattern 500. Forexample, the angle between the portions 506 a and 506 b in the centralregion 502 is between 80 degrees and 90 degrees and the angle betweenthe portions 506 b and 506 c in the edge region 504 is less than 45degrees. As another example, an angle between a portion 510 a of theLissajous pattern 500 in the edge region 504 and a portion 510 b of theLissajous pattern 500 in the edge region 504 is less than 45 degrees.

The Lissajous pattern 500 has an origin 512 at which there is zeroangular displacement of the beam 110 with respect to the xyz co-ordinatesystem of the projector 104. The Lissajous pattern 500 starts at theorigin 512 and ends at the origin 512 and then repeats in a continuousway.

In an embodiment, the Lissajous pattern 500 is represented as:x=A sin(ω₁ t+δ)  (equation 1)along the X-axis of the MEMS mirror 112, and asy=B sin(ω₂ t)  (equation 2),along the Y-axis of the MEMS mirror 112, where ω₁=2π t f1 and ω₂=2π tf2, t is time, A is an amplitude of the Lissajous pattern 500 along theX-axis of the MEMS mirror 112, and B is an amplitude of the pattern 500along the Y-axis of the MEMS mirror 112, and δ is a phase differencebetween the Lissajous pattern 500 along the X-axis of the MEMS mirror112 compared to the Y-axis of the MEMS mirror 112.

In one embodiment, the phase δ is not needed. So, the equation (1)becomesx=A sin(ω₁ t)  (equation 3).In an embodiment in which f1 is greater than f2, a repeat frequency F atwhich a frame of the Lissajous pattern 500 is repeated is equal to adifference between f1 and f2. A value of f1 and/or a value of f2determines a density of the Lissajous pattern 500.

FIG. 6A is a diagram of an embodiment to illustrate a measured location602 of the photosensor 105 a with respect to an actual location 604 ofthe photosensor 105 a to determine the location of the beam hit in theLissajous pattern 500. The measured location 602 of the photosensor 105is determined with reference to the origin of the xyz co-ordinate systemof the projector 104. The Lissajous pattern 500 of the beam 110 is shownas being incident on the photosensor 105 a and has a Lissajous axis 606,which provides a direction followed by the Lissajous pattern 500. Adiameter of the Lissajous pattern 500 is d1, e.g., 10 millimeters, 5millimeters, 8 millimeters, etc. Instead of the actual location 604, theprocessor 302 determines the location of the photosensor 105 a withrespect to the origin of the xyz co-ordinate system of the projector 104to be anywhere along an axis 608 orthogonal to the Lissajous axis 606.For example, the processor 302 determines the location of thephotosensor 105 a with respect to the origin of the xyz co-ordinatesystem of the projector 104 to be at a point at the measured location602. The point at the measured location 602 is at an intersection of theLissajous axis 606 and the axis 608. The axis 608 extends via the actuallocation 604 at which the beam hit is detected. In this way, a readingwill have accuracy along a sweep of the beam 110 but its accuracyorthogonal to the beam 110 depends upon the diameter d1 of the beam 110and a size, e.g., a diameter, etc., of the photosensor 105 a. As shown,a diameter of the photosensor 105 a is smaller than the diameter d1 ofthe Lissajous pattern 500 followed by the beam 110.

FIG. 6B is a diagram of an embodiment to illustrate that the location ofthe beam hit is determined using an intersection of the diameter d1 ofthe portion 506 a and the diameter d1 of the portion 506 b of theLissajous pattern 500 when there is no movement in the photosensor 105 abetween the illumination of the photosensor by each of the Lissajouspattern sweeps shown. The location of the beam hit is the actuallocation 604, which is at the intersection of the diameter d1 of theportion 506 a and the diameter d1 of the portion 506 b. The photosensor105 a generates a detected beam hit signal at a time t1 during theportion 506 a of the Lissajous pattern 500 and generates anotherdetected beam hit signal at a time t2 during the portion 506 b of theLissajous pattern 500. The portion 506 a traverses in a directionangular, e.g., at a 90 degree angle, at an angle between 80 and 95degrees, etc., to a direction in which the portion 506 b traverses.Moreover, there is no movement of the photosensor 105 a when theLissajous pattern 500 traverses in the angular directions. The locationof the beam hit is determined when the photosensor 105 a detects thebeam hit multiple times, e.g., the times t1 and t2, etc., during theframe, e.g., one round of execution, etc., of the Lissajous pattern 500.

In one embodiment, when the Lissajous pattern 500 and a size of thephotosensor 105 a are matched, e.g., same diameter, diameters within apre-determined range from each other, etc., a first detected beam hitsignal is obtained during the frame when the portion 506 a sweeps, e.g.,during a first sweep, etc., across the photosensor 105 a and a seconddetected beam hit signal is obtained during the frame when the beam 110sweeps, e.g., during a second sweep, etc., approximately orthogonally tothe first sweep. The second sweep occurs when the portion 506 b sweepsacross the photosensor 105 a. The clock source of the HMD 102 measures atime at which the first detected beam hit signal is generated andmeasures a time at which the second detected beam hit signal isgenerated. The processor 302 determines from the times associated withthe first and second detected beam hit signals the orientation of thephotosensor 105 a with respect to the origin of the xyz co-ordinatesystem of the projector 104. For example, the processor 302 determines acorresponding first orientation of the beam 110 with respect to theorigin of the xyz co-ordinate system of the projector 104 based on thetime at which the first detected beam hit signal is generated. Moreover,the processor 302 determines a corresponding second orientation of thebeam 110 with respect to the origin of the xyz co-ordinate system of theprojector 104 based on the time at which the second detected beam hitsignal is generated. The processor 302 calculates a statistical value,e.g., an intersection of the d1 uncertainty between the first and secondorientations, etc., to determine an orientation of the beam 110 withrespect to the origin of the xyz co-ordinate system of the projector104. In one embodiment, the orientation of the beam 110 that isdetermined by the processor 302 is the same as that of the photosensor105 a with respect to the origin of the xyz co-ordinate system of theprojector 104.

FIG. 6C is a diagram of an embodiment to illustrate that the location ofthe beam hit is determined using an intersection of the diameter d1 ofthe portion 506 a and the diameter d1 of the portion 506 b of theLissajous pattern 500 and the location is corrected when there ismovement in the photosensor 105 a. Between a movement from the portion506 a to the portion 506 b of the Lissajous pattern 500, the photosensor105 a moves from a position P1 to a position P2 by an amount 610. At theposition P1, the photosensor 105 a generates a detected beam hit signalat a time ta during the portion 506 a of the Lissajous pattern 500 andat the position P2, the photosensor 105 a generates another detectedbeam hit signal at a time tb during the portion 506 a of the Lissajouspattern 500. An intersection of the diameter d1 when the photosensor 105a is at the position P1 and the diameter d1 when the photosensor 105 ais at the position P2 is an error location of the photosensor 105 a withrespect to the origin of the xyz co-ordinate system of the projector104. The error location is corrected by the processor 302 based on themovement 610 to remove an error 614 in the error location 612 to achievethe location of the photosensor 105 a with respect to the origin of thexyz co-ordinate system of the projector 104. The movement 610 ispredicted by the processor 302 based on previously stored movement ofthe photosensor 105 a that is recorded in the memory device 304 of thecomputing device 107 and may be the result of sensor fusion with othermeasurement sensors, such as, inertial measurement units (IMUs), e.g.,accelerometers and gyroscopes, etc.

In an embodiment, the method that is described above with reference toFIG. 6B or FIG. 6C in which the intersection of the diameters d1 isused, is used to determine the location of the photosensor 105 a withrespect to the origin of the xyz co-ordinate system of the projector 104when the photosensor 105 a encounters the beam 110 in the edge region504 (FIG. 5) of the Lissajous pattern 500.

FIG. 6D shows the portions 510 a and 510 b of the Lissajous pattern 500to illustrate a determination of the location of the photosensor 105 awith respect to the origin of the xyz co-ordinate system of theprojector 104 when the photosensor 105 a is stationary and encountersthe beam 110 in the edge region 504 (FIG. 5) of the Lissajous pattern500. A location of the photosensor 105 a that detects the beam 110 isdetermined by the processor 302 to be at an intersection 620 of aninterval ia of the portion 510 a and an interval ib of the portion 510b. For example, at a time tc at which the photosensor 105 a detects theportion of the interval ia of the beam 110, the processor 302 calculatesa curved trajectory of the Lissajous pattern segment 510 a and similarlyat a time td at which the photosensor 105 a detects a portion of theinterval ib of the beam 110, the processor 302 calculates a curvedtrajectory of the Lissajous pattern segment 510 b. The measured locationof the photosensor 105 a with respect to the origin of the xyzco-ordinate system of the projector 104 is calculated by the processor302 by intersecting the two measured curved trajectories, e.g.,segments, etc.

FIG. 7A is a diagram of an embodiment of a system 700 for illustratingthe method of triangulation to determine the position of the photosensor105 a with respect to the projector 104. The system 700 includes theprojector 104 and a projector 702, e.g., two projectors instead of one,etc. The projector 702 is the same as the projector 104 in structure andfunction. Projectors 104 and 702 are fixed in a rigid and known topologywith respect to one another in order to provide a known baseline fortriangulation. A direction of the beam 110 emitted from the projector104 is at an angle θa with respect to the z-axis of the projector 104and a direction of a beam 704, e.g., laser beam, visible light beam,infrared light beam, invisible light beam, etc., emitted from theprojector 702 is at an angle θb with respect to an z-axis of theprojector 702, and both the angles θa and θb are determined by theprocessor 302 of the computing device 107 by applying the operation 458(FIG. 4C) of determining the orientation. The projector 702 includes anMEMS mirror, which is the same as the MEMS mirror 112 of the projector104. The z-axis of the projector 702 is perpendicular to an x-axis ofthe projector 702 and both the z and x axes of the projector 702 areperpendicular to a y-axis of the projector 702.

The processor 302 applies the method of triangulation to the angles θaand θb to determine an intersection (or closest points of two lines) ofthe beam 110 with the beam 704, and determines that a point of theintersection is a position of the photosensor 105 a, of the HMD 102,that detects the beams 110 and 704. The position of the photosensor 105a is measured with respect to the origin of the xyz co-ordinate systemof the projector 104 and/or of an xyz co-ordinate system of theprojector 702. The xyz co-ordinate system of the projector 702 includesthe x-axis of the projector 702, the y-axis of the projector 702, andthe z-axis of the projector 702.

In an embodiment, the projectors 102 and 702 time division multiplexgeneration of the beams 110 and 704 to identify one of the projectors104 and 702 from the corresponding one of the beams 110 and 704 detectedby the photosensor 105 a of the HMD 102. The projector 104 emits thebeam 110 at a frequency, e.g., 60 Hz, etc., which is a frequency of theLissajous pattern scan, e.g., a frequency at which the Lissajous patternis repeated by the MEMS mirror 112, etc. Then after the pattern isexecuted by the projector 104, the projector 104 stops emitting the beam110, and the projector 702 emits the beam 704 at the same frequency atwhich the beam 104 was emitted to form the same pattern on the MEMSmirror of the projector 702 as that formed by the beam 110. Then, afterthe pattern is formed by the beam 704, the projector 702 stops emittingthe beam 704, and the projector 104 emits the beam 110 at the samefrequency at which the beam 704 was emitted. This is how the projectors104 and 702 time division multiplex.

In one embodiment, instead of time division multiplexing at thefrequency of 60 Hz, the projectors 104 and 702 time division multiplexat a frequency of thousands of hertz, e.g., a frequency at which theMEMS mirror 112 moves with respect to the resonant frequency of the MEMSmirror 112.

In an embodiment, the processor of the projector 104 sends a controlsignal to the processor 302 of the computing device 107 via the link 111indicating that the beam generator 108 of the projector 104 has stoppedemitting the beam 110 and a processor of the projector 702 sends acontrol signal to the processor 302 via a link 706, same in structureand function as the link 111, indicating that a beam generator of theprojector 702 has stopped emitting the beam 704. Moreover, the processorof the projector 104 sends a control signal to the processor 302 of thecomputing device 300 via the link 111 indicating that the beam generator108 of the projector 104 has started emitting the beam 110 and theprocessor of the projector 702 sends a control signal to the processor302 via the link 706 indicating that the beam generator of the projector702 has started emitting the beam 704. The processor of the projector104 is coupled to the beam generator 108 and the processor of theprojector 702 is coupled to the beam generator of the projector 702. Thecontrol signals sent by the projector 104 to the processor 302 includean identity of the projector 104 and the control signals sent by theprojector 702 to the processor 302 include an identity of the projector702. Upon receiving the control signals from the projector 104, theprocessor 302 identifies the projector 104 from the identity of theprojector 104. Similarly, upon receiving the control signals from theprojector 702, the processor 302 identifies the projector 702 from theidentity of the projector 702.

In an embodiment, the processor 302 of the computing device 107 sendscontrol signals to the processors of the projectors 104 and 702 to turnon the beam generator 108 of the projector 104 and to turn on the beamgenerator of the projector 702, and sends control signals to theprocessors of the projectors 104 and 702 to turn off the beam generator108 of the projector 104 and to turn on the beam generator of theprojector 702 to time division multiplex generation of the beams 110 and704. For example, the computing device 107 sends a control signal to theprojector 104 to turn on the projector 104 at a time, e.g., during aclock cycle, during a rising edge of the clock cycle, during a fallingedge of the clock cycle, etc., and at the same time sends a controlsignal to the projector 702 to turn off the projector 702. As anotherexample, the computing device 107 sends a control signal to theprojector 702 to turn on the projector 702 at a time, e.g., during aclock cycle, during a rising edge of the clock cycle, during a fallingedge of the clock cycle, etc., and at the same time sends a controlsignal to the projector 104 to turn off the projector 104. A processorof a projector receives a control signal to turn on the projector, andsends the control signal to a driver, e.g., one or more transistors,etc., of a beam generator of the projector so that the driver turns onand generates a current signal to drive a beam source, e.g., a lightemitting diode (LED) lamp, an incandescent lamp, a neon lamp, agas-discharge lamp, a xenon arc lamp, a laser beam emitter, anelectromagnetic radiation emitter, a gas laser source, a chemical lasersource, a solid-state laser source, a fiber laser source, a photoniccrystal laser source, a semiconductor laser source, a diode, etc., ofthe beam generator. Upon receiving the current signal from the driverfor turning on the beam source, the beam source emits a beam of light.Similarly, the processor of the projector receives a control signal toturn off the projector, and sends the control signal to the driver ofthe projector so that the driver turns off and discontinues generatingthe current signal. When the current signal is discontinued to begenerated, the beam source ceases to emit the beam of light.

The computing device 107 is coupled to the projector 702 via the link706 and sends control signals to the projector 702 via the link 706.Similarly, the projector 702 communicates with the computing device 107via the link 706. Moreover, the computing device 107 sends controlsignals to the projector 104 via the link 111. Similarly, the projector104 communicates with the computing device 107 via the link 111.

It should be noted that in some embodiments, the methods described abovewith reference to FIGS. 4A-4E are applicable to the projector 702 andthe HMD 102 instead of the projector 104 and the HMD 102. For example,the method of FIG. 4A is used to determine the orientation of the beam704 with respect to the xyz co-ordinate system of the projector 702 andthe method described with reference to FIGS. 4B and 4C is used todetermine the orientation of the beam 704 with respect to the xyzco-ordinate system of the projector 702 and the position of thephotosensor 105 a with respect to the xyz co-ordinate system of theprojector 702.

FIG. 7B is a diagram of an embodiment of a system 750 for illustratinganother method to determine a position of the photosensor 105 a withrespect to the projector 104. When the photosensors 105 a, 105 b, and105 c are hit by the beam 110, the processor 302 of the computing device107 receives timings of generation of the detected beam hit signals fromthe processor of the HMD 102. The timings are measured by the clocksource of the HMD 102 and obtained by the processor of the HMD 102 fromthe clock source of the HMD 102. The processor 302 determinesorientations of the photosensors 105 a, 105 b, and 105 c from thetimings in a manner similar to determining the orientation in theoperation 458 (FIG. 4C). The processor 302 applies a method, such as,Perspective-n-Point, etc., to estimate the pose of the HMD 102 withrespect to the origin of the xyz co-ordinate system of the projector104.

FIG. 8A is a diagram of an embodiment of the system 700 to illustrate amodulation of the beams 110 and 704 to identify the projectors 104 and702. A modulator, e.g., a digital modulator, etc., of the projector 104is coupled to the beam source of the beam generator 108 of the projector104 to modulate a beam that is emitted by the beam source of theprojector 104 with a first bit code, e.g., a bit pattern, a sequence ofL bits, etc., where L is an integer greater than two. For example, thebeam emitted by the beam source of the beam generator 108 is overlaidwith a sequence of bits so that the pulse of the detected beam hitsignal generated by the photosensor 105 a is overlaid with a firstsequence of L bits. The beam emitted by the beam source of the projector104 is modulated to generate the beam 110.

Similarly, a beam emitted by a beam source of the beam generator of theprojector 702 is overlaid with a sequence of bits so that a pulse of adetected beam hit signal generated by the photosensor 105 a upondetecting the beam 704 is overlaid with a second bit code, e.g., a bitpattern, a second sequence of L bits, etc. The second bit code has adifferent combination of bits than the first bit code to distinguish thebeam 704 from the beam 110. For example, the first sequence has a bitcombination of 0110011 and the second sequence has a bit combination of1001100. A modulator, e.g., a digital modulator, etc., of the projector702 is coupled to the beam source of the beam generator of the projector702 to modulate the beam that is emitted by the beam source of theprojector 702 with a sequence of bits so that a pulse of a detected beamhit signal emitted by the photosensor 105 a has overlaid thereon thesecond bit code. Examples of the first and second bit codes are providedbelow.

In an embodiment, both the beams 110 and 704 that are generated afterthe modulation with the corresponding sequences of bits are emitted atall times. For example, the beam generator 108 of the projector 104 andthe beam generator of the projector 702 continuously emit theirrespective beams. There is no time division multiplexing by theprojectors 104 and 702 to generate the beams 110 and 704. A collisionduring which the two beams 110 and 704 hit the photosensor 105 a at thesame time is rare and these cases may be discounted as error cases.

In one embodiment to avoid the collision, both the beams 110 and 704that are generated after the modulation with the corresponding sequencesof bits are emitted at different times. For example, the beam generator108 of the projector 104 emits the beam 110 when the beam generator ofthe projector 702 does not emit the beam 704 and the beam generator 108of the projector 104 does not emit the beam 110 when the beam generatorof the projector 702 emits the beam 704. There is time divisionmultiplexing by the projectors 104 and 702 to generate the beams 110 and704.

When the beams 110 and 704 are incident on the photosensor 105 a, thephotosensor 105 a of the HMD 102 detects the beam 110 to generate afirst detected beam hit signal and detects the beam 704 to generate asecond detected beam hit signal. The first and second detected beam hitsignals generated by the photosensor 105 a are provided from thephotosensor 105 a to a demodulator of the HMD 102. The demodulator ofthe HMD 102 demodulates the first detected beam hit signal to extractthe first sequence of L bits from the first detected beam hit signal anddemodulates the second detected beam hit signal to extract the secondsequence of L bits from the second detected beam hit signal. Theprocessor 302 of the computing device 107 receives the first and secondsequences of L bits from the demodulator of the HMD 102 via the link 109(FIG. 1A), determines that the first sequence has a differentcombination, e.g., pattern, etc., of bits compared to the secondsequence of bits and determines that the first detected beam hit signalis generated from the beam 110 generated the projector 104 and thesecond detected beam hit signal is generated from the beam 704 generatedthe projector 702. A correspondence, e.g., a mapping, an association,etc., between the projectors 110 and 702 and the first and secondsequences is stored in the memory device 304 of the computing device107. For example, an identity of the projector 104 is in the same row asthat of the first sequence of bits and an identity of the projector 702is in the same row as that of the second sequence of bits. Thecorrespondence is accessed by the processor 302 to determine whether thefirst sequence is generated by the projector 104 or the projector 702and whether the second sequence is generated by the projector 104 or bythe projector 702. As such, the projectors 104 and 702 are identified bythe processor 302.

In one embodiment, a frequency of a modulating signal, generated by amodulator, having a unique bit code is high, e.g., 100 MHz, 90 MHz, afrequency between 90 MHz and 110 MHz, a frequency of 1 GHz or higher,etc. An example of the modulating signal is a sequence of bits, e.g.,the first sequence of bits, the second sequence of bits, etc.

In an embodiment, a modulator of a projector is computer softwareprogram that is executed by a processor of the projector to modulate abeam that is generated by a beam source of the projector. For example,the modulator of the projector 104 is a computer software routine thatis executed by the processor of the projector 104 and the modulator ofthe projector 702 is a computer software routine that is executed by theprocessor of the projector 702. In one embodiment, a demodulator of theHMD 102 is computer software program that is executed by the processorof the HMD 102 to demodulate the detected beam hit signal that isgenerated the photosensor 105 a of the HMD 102.

In one embodiment, a modulator is an electrical circuit and ademodulator is an electrical circuit.

In an embodiment, in a manner similar to determining the orientation ofthe photosensor 105 a with respect to the origin of the xyz co-ordinatesystem of the projector 104 during the frame of the pattern, multipledetected beam hit signals generated by remaining photo sensors, e.g.,the photosensors 105 b and 105 c, etc., of the HMD 102 are used todetermine orientations of the photosensors of the HMD 102 with respectto the xyz co-ordinate system of the projector 104. The orientations ofthe remaining photosensors of the HMD 102 with respect to the xyzco-ordinate system of the projector 702 are determined by the processor302. Triangulation or Perspective-n-Point methods are applied by theprocessor 302 to the orientations of the photosensors, e.g., thephotosensor 105 a, the photosensor 105 b, and the photosensor 105 c,etc., of the HMD 102 with respect to the xyz co-ordinate system of theprojector 104 and the xyz co-ordinate system of the projector 702 todetermine an orientation and a position of the HMD 102 with respect tothe xyz co-ordinate system of the projector 104 or the xyz co-ordinatesystem of the projector 702.

In an embodiment, a motion of the HMD 102 is taken into account by theprocessor 302 to determine an orientation and a position of the HMD 102with respect to the xyz co-ordinate system of the projector 104 or thexyz co-ordinate system of the projector 702.

In the case of the multiple projectors, e.g., the projectors 104 and702, etc., in an embodiment, beam collision occurs between beams emittedby the projectors, e.g., the beams 104 and 704, etc. The beam collisionoccurs when the photosensor 105 a is simultaneously hit by two or moreprojector beams, e.g., the beams 104 and 704, etc. Chances of thecollision increase with an increase in a number of the projectors, withan increase in diameters of the beams emitted by the projectors, andwith a decrease in distance between the HMD 102 and the projectors. Thebeam collision is modeled in a simulation. Chances of the beam collisionare reduced by using different light frequencies of the beams and byusing multiple photo sensors with notch filters. For example, afrequency of the beam 110 is different from a frequency of the beam 704.

In one embodiment, the collisions are reduced by switching the beamsemitted by the projectors off at a perimeter, e.g., the edge region 504,etc., of the Lissajous pattern 500 (FIG. 5) where the Lissajous pattern500 is more dense compared to the central region 502 (FIG. 5). Forexample, the beam generator 108 is turned off, e.g., powered off, etc.,at a time the beam 110 reaches the edge region 504 from the centralregion 502 on the Lissajous pattern 500 and the beam generator of theprojector 702 is turned off at a time the beam 704 reaches the edgeregion 504 from the central region 502 on the Lissajous pattern 500. Inan embodiment, a processor of a projector sends a control signal to apower source, e.g., a battery, an alternating current supply, a powergenerator, etc., of the projector to turn off the power source at a timethe processor determines that a beam reaches the edge region 504 fromthe central region 502 on the Lissajous pattern 500. The power source iscoupled to the processor. The power source is coupled to a beamgenerator of the projector to provide power to the beam generator.

In an embodiment, the projector 104 is turned on at a time the beam 110reaches the central region 502 from the edge region 504 on the Lissajouspattern 500 and the projector 702 is turned on, e.g., powered on, etc.,at a time the beam 704 reaches the central region 502 from the edgeregion 504 on the Lissajous pattern 500. In an embodiment, a processorof a projector sends a control signal to a power source of the projectorto turn on the power source at a time the processor determines that abeam reaches the central region 502 from the edge region 504 on theLissajous pattern 500.

In one embodiment, a first modulation frequency, e.g., a frequency withwhich the beam 110 is modulated with a bit sequence, etc., of the beam110 is different from a second modulation frequency, e.g., a frequencywith which the beam 704 is modulated with a bit sequence, etc., of thebeam 704. The processor 302 receives first and second detected beam hitsignals that represent the two modulation frequencies, and applies aFourier transformation to convert the first and second detected beam hitsignals from a time domain to a frequency domain to extract the twodifferent modulation frequencies from the detected beam hit signals. Theprocessor 302 identifies from the first modulation frequency and acorrespondence stored between the first modulation frequency and anidentity of the projector 104 that the first detected beam hit signal isgenerated from the beam 110 emitted from the projector 104. Similarly,the processor 302 identifies from the second modulation frequency and acorrespondence stored between the second modulation frequency and anidentity of the projector 702 that the second detected beam hit signalis generated from the beam 704 emitted from the projector 702.

In an embodiment, a first filter that filters in the first frequency isconnected to the photosensor 105 a and a second filter that filters inthe second frequency is connected to another photosensor, e.g., thephotosensor 105 b, etc., that is in close proximity to the photosensor105 a. The first filter receives the first detected beam hit signalhaving the first frequency from the photosensor 105 a to output a firstfiltered signal and the second filter receives the second detected beamhit signal having the second frequency from the other photo sensor tooutput a second filtered signal. The processor 302 receives the firstfiltered signal from the first filter via the link 109 (FIG. 1A) and thesecond filtered signal from the second filter via the link 109. Upondetermining that the first filtered signal is received from the firstfilter and that the second filtered signal is received from the secondfilter, the processor 302 identifies that the first filtered signal isgenerated based the beam 110 emitted from the projector 104 and that thesecond filtered signal is generated based the beam 704 emitted from theprojector 702. The first and second filters are located within the HMD102.

FIG. 8B is a diagram of an embodiment of a pulse 810 of the detectedbeam hit signal that is generated by the photosensor 105 a upondetecting the beam hit from the beam 110. The pulse 810 as illustratedis overlaid with a sequence of L bits.

In an embodiment, K independent bit sequences with globally unique M-bitwindows are generated with each of the K independent bit sequenceshaving a length L, where K is a number of the projectors 104 and 702that are uniquely identified, and L is the number of bits before a bitpattern for that sequence repeats itself. For example, the photosensor105 a generates the detected beam hit signal in which each pulse of thedetected beam hit signal is modulated by the modulator of the projector104 to be overlaid with the L bit sequence. As another example, thephotosensor 105 a generates the detected beam hit signal in which eachpulse of the detected beam hit signal is modulated by the modulator ofthe projector 702 to be overlaid with the L bit sequence. Within each ofthese sequences of L bits, any sub window of M bits is unique within thesequence and also globally unique across all of the K independent bitsequences. For example, the photosensor 105 a generates the detectedbeam hit signal in which a first pulse of the detected beam hit signalis modulated by the modulator of the projector 104 to be overlaid with afirst L bit sequence and in which a second pulse of the detected beamhit signal is modulated by the modulator of the projector 104 to beoverlaid with a second L bit sequence. The second L bit sequence has adifferent M bit sequence of bits than an M bit sequence of bits withinthe first L bit sequence and than any other M bit sequences in thesecond L bit sequence. The photosensor 105 a generates the detected beamhit signal in which a third pulse of the detected beam hit signal ismodulated by the modulator of the projector 702 to be overlaid with athird L bit sequence and in which a fourth pulse of the detected beamhit signal is modulated by the modulator of the projector 702 to beoverlaid with a fourth L bit sequence. The fourth L bit sequence has adifferent M bit sequence of bits than any remaining M bit sequenceswithin the fourth L bit sequence, any M bit sequence of bits within thethird L bit sequence, than any M bit sequence within the first L bitsequence, and than any M bit sequence within the second L bit sequence.

In an embodiment, any M bit sequence is decoded by a demodulator. Toachieve the decode, a modulator modulates a beam so that any M bitsequence has less than a pre-determined number of instances of the samebit and at least one of 010 and 101 switch binary pattern is introducedin any M bit sequence to facilitate identification of the binary patternwith error resilience.

In an embodiment, a modulator modulates a beam so that each M bitsequence is constrained to be different from any another M bit sequenceby at least 2 bits. This constraint allows for error resilience.

In one embodiment, a bit rate applied by the modulator of the projector104 to overlay a bit sequence onto a beam is determined based on amaximum possible distance of the HMD 102 from the projector 104, and thefrequencies f1 and f2. For example, the modulator of the projector 104applies a high data rate to overlay a bit sequence onto a beam generatedby the beam source of the beam generator 108 so that when the beam 110is moving quickly to scan the space, and the photosensor 105 a is placedat a maximum defined distance, there is enough time for the photosensor105 a to generate a detected beam hit signal containing the M bitsequence. An example of the maximum defined distance is a distancebetween two opposite walls of a room in which the HMD 102 is located.Another example of the maximum defined distance is a distance betweenthe projector 104 and a wall, which is located on a side of a room wherethe HMD 102 is located compared to another side of the room where theprojector 104 is located. It should be noted that movement of the beam110 in the space is dependent upon the frequencies f1 and f2 of movementof the MEMS mirror 112.

In an embodiment, the beam 110 makes a glancing hit on the photosensor105 a such that a time to modulate a beam generated by the beam sourceof the projector 104 is reduced. The reduction in the time reduceschances that the M bit sequence is embedded within a pulse of a detectedbeam hit signal generated by the photosensor 105 a. In this embodiment,the M bit sequence is arranged by the modulator of the projector 104 tobe across a radius of the beam generated by the beam source of theprojector 104 and not across a diameter of the beam generated by thebeam source of the projector 104. A bit rate at which the beam generatedby the beam source of the projector 104 is modulated by the modulator ofthe projector 104 is determined by the modulator based upon the maximumdefined distance and the radius of the beam at the maximum possibledistance. The bit rate is determined such that each pulse of thedetected beam hit signal generated by the photosensor 105 a has the Mbit sequence.

In one embodiment, the photosensor 105 a is modeled as a disc of radiusr, and the detected beam hit signal is generated by the photosensor 105a when the beam 110 traverses slightly to a side of the disc's center,sometimes referred to herein as a grazing beam hit. An example of theradius of the disc is a radius of a lens of the photosensor 105 a thatis exposed to the beam 110. Furthermore, a traversal distance of thebeam 110 is equal to the radius of the disc. This allows a reasonablecoverage or a reasonable number of grazing beam hits. When the disc isat a center of the pattern, the beam 110 sweeps across at its maximumspeed due to a nature of sinusoidal motion of the MEMS mirror 112. Atthe maximum speed, over the radius of the disc, the modulator of theprojector 104 modulates the beam generated by the beam source of theprojector 104 with the M bit sequence so that the photo sensor 105 agenerates the detected beam hit signal having the M bit sequence. Assuch, the radius of the disc and the maximum speed defines a bit-rate ofthe modulation performed by the modulator of the projector 104.

In one embodiment, a bit rate applied by a modulator of a projector tooverlay a bit sequence onto a beam generated by a beam source of theprojector is a high data rate. For example, when f1=21,060 Hz, andf2=21,000 Hz, and the pattern, e.g., a square pattern, etc., has anamplitude of 60 degrees left to right displacement, e.g., +/−30 degreesfrom a center of the square pattern, and the photosensor 105 a has an 1millimeter diameter, a radius of the beam 110 is 5 millimeters at a 3meter distance from the projector 104, and M=10 bit sequence is used,two disjoint sequences of length 61 bits are identified by the processor302 after demodulation. The two sequences are L=61 bit long and are M=10bit sequences. The L bit sequences both repeat and any M=10 bit sequenceis unique within each L bit sequence and across both the L bitsequences. In addition, each M bit sequence is constrained by themodulator to not have a consecutive sequence of the same bits longerthan two, e.g., no 000 or no 111, etc., and is constrained by themodulator to include at least one bit flip pattern, e.g., 101 or 010,etc. An example of a first one of the two L bit sequences is0010010010100100110010010110010011010010101010011010101101101 and anexample of a second one of the two L bit sequences is0010101101001101101010010110101010101100110101101011001011011. Anexample of the bit rate applied by a modulator of a projector tomodulate a beam is 370 megabits per second (Mbps). Another example ofthe bit rate applied by a modulator of a projector to modulate a beam is500 Mbps.

In an embodiment, when a demodulator decodes at least 10 bits of a 61bit sequence of the length L and there is a synchronization between theclock signal generated by the HMD 102 and the clock signal generated bythe projector 104, the M bit sequence of the L bit sequence is used toidentify an exact location within the 61 bit sequence to further providean accurate time measurement for a beam hit.

In one embodiment, since the M bit sequence is unique across the L bitsequences of the K projectors, a projector that generates a beam inwhich the M bit sequence is encoded in is identified. For example, ademodulator, e.g., decoder, etc., of the HMD 102 decodes a detected beamhit signal to extract at least M bits of the L sequence of bits. The Lsequence of bits is provided from the demodulator via the link 109 (FIG.1A) to the processor 302 of the computing device 107. The processor 302identifies from the L bits the M bit sequence that is unique to all Lbit sequences of the projectors 104 and 702. The processor 302 comparesthe M bit sequence with a sequence pre-stored in the memory device 304to determine whether there is a match between the two sequences, andupon determining so, determines that the M bit sequence is overlaid on apulse of the detected beam hit signal generated when the projector 104emits the beam 110. A correspondence, e.g., one-to-one relationship,etc., between a projector and the M bit sequence is stored in the memorydevice 304.

In an embodiment, longer M bit sequences allow for longer L bitsequences and better tolerances of time synchronization and the longer Mbit sequences mean a larger value of M and consequently a higher bitrate.

In one embodiment, the modulator of the projector 104 modulates the beamgenerated by the beam source of the projector 104 with an M-sequencewindow of M bit sequences. Within the M-sequence window, any M bitsequence is unique. When the photosensor 105 a generates the detectedbeam hit signal having the pulse that has one or more of the M bitsequences, the processor 302 determines a location of one of the M bitsequences within the M-sequence window. The clock source of the HMD 102measures a time t_(s) at which the one of the M bit sequences startswithin the M-sequence window. The one of the M bit sequences is closestamong all of the M bit sequences within the M-sequence window to acenter of a peak of the pulse 810 of the detected beam hit signal. Theclock source of the HMD 102 further measures an offset time, t_(off),which is a time difference between the time t_(s) and a time at which acenter of the one of the M bit sequences occurs within the pulse. Thevalues t_(s), t_(off) and the one of the M bit sequences are sent fromthe processor of the HMD 102 to the processor of the projector 104 viathe link 117, e.g., Bluetooth link, an RF link, etc. The processor ofthe projector 104 based on the time t_(s) finds a bit position withinthe M-sequence window that the projector 104 transmitted. The processorof the projector 104 searches forward and backwards in the M-sequencewindow from the bit position to find a location of the one of the M bitsequences and determines from a memory device of the projector 104 atime t_(a) at which the one of the M bit sequences is sent from theprojector 104 as embedded within the beam 110. The time t_(a) ismeasured by the clock source of the projector 104 and is stored by theprocessor of the projector 104 in the memory device of the projector104. The processor of the projector 104 then corrects the time t_(s) tothe time t_(a) and sends the time t_(a) to the processor 302. Theprocessor 302 determines based on the time t_(a) the orientation of thephotosensor 105 a with respect to the xyz co-ordinate system of theprojector 104 in a manner described in the operations 456 and 458 (FIGS.4B and 4C). For example, the time t_(a) is used as a time of a hit todetermine the orientation of the photosensor 105 a with reference to theorigin of the xyz co-ordinate system of the projector 104.

In an embodiment, the processor of the projector 104 calculates a timedifference t_(s)−t_(a) in the times t_(s) and t_(a). The time differencet_(s)−t_(a) is a clock error between the clock signal of the HMD 102 andthe clock signal of the projector 104, and the processor of theprojector 104 sends a signal indicating the time difference to theprocessor of the HMD 102. The processor of the HMD 102 sends a controlsignal to the clock source of the projector 104 to remove the timedifference to synchronize the clock signal of the projector 104 with theclock signal of the HMD 102.

In an embodiment, the processor of the projector 104 computes a logicoffset, which is a time difference t_(a)−t_(off), to obtain t_(r), whichis a time at which the beam 110 hit a center of the disc of thephotosensor 105 a. The time t_(r) is sent from the processor of theprojector 104 to the processor 302 via the link 111. Upon receiving thetime t_(r), the processor 302 applies the operations 456 and 458 (FIGS.4B and 4C) determines the orientation of the photosensor 105 a withrespect to the xyz co-ordinate system of the projector 104. For example,the time t_(r) is used as a time of a hit to determine the orientationof the photosensor 105 a with reference to the origin of the xyzco-ordinate system of the projector 104.

FIG. 8C is a diagram to illustrate than the L bit sequence for aprojector repeats many times along the Lissajous pattern 500. Eachsequence S1, S2, S3, S4, S5, S6, and S7 is an L bit sequence.

In an embodiment, each sequence S1, S2, S3, S4, S5, S6, and S7 repeatsany other number of times, e.g., 150 times, 100 times, 200 times, etc.during a sweep from a diagonal corner, e.g., a diagonal left corner 830,to an oppositely-located diagonal corner, e.g., a diagonal right corner832, e_(tc.), of the Lissajous pattern 500. In one embodiment, theLissajous pattern 500 has any other number, e.g., eight, nine, ten,etc., of L bit sequences than that shown in FIG. 8C.

FIG. 8D is a diagram of an embodiment of a correlation filter 864 thatis applied by the processor 302 of the computing device 107 to detectthe L bit sequence. The correlation filter 864 is applied by theprocessor 302 when a detected beam hit signal on which a bit sequence isoverlaid is generated by the photosensor 105 a and the demodulator ofthe HMD 102 demodulates the detected beam hit signal to provide the bitsequence to the processor 302. The correlation filter 864 is stored inthe memory device 302. In one embodiment, a correlation filter is apre-stored sequence of L bits. When the processor 302 determines that amatch between the correlation filter 864 and a consecutive sequence ofbits within the bit sequence received from the demodulator of the HMD102 has occurred, the processor 302 determines that the L bit sequenceis found. As illustrated in FIG. 8D, the processor 302 determines that aportion of less than M bits of the L bit sequence is found at 5 places,labeled, as A, B, C, D, and E in a bit sequence 862 received from thedemodulator.

In one embodiment, upon determining that a number of matches between thecorrelation filter 864 and the bit sequence 862 exceeds one, or anotherpre-determined number, the processor 302 disregards the matches sincenot enough bits were decoded to uniquely identify the L bit sequence.

FIG. 8E is a diagram of an embodiment illustration application of athreshold to the correlation filter 864. The processor 302 determines aunique match between a consecutive sequence of bits within a bitsequence 872 received from the demodulator of the HMD 102 via the link109 and the correlation filter 864, and determines that the L bitsequence is received within a detected beam hit signal. As the number ofbits in a demodulated sequence increases above M, a probability that theL bit sequence is detected within a beam hit signal increases. As shownin FIG. 8E, a match occurs at a place labeled A in the sequence 864.

The correlation filter 864 is applied by the processor 302 when adetected beam hit signal on which a bit sequence is overlaid isgenerated by the photosensor 105 a and the demodulator of the HMD 102demodulates the detected beam hit signal to provide the bit sequence tothe processor 302. The correlation filter 864 is stored in the memorydevice 302. In one embodiment, a correlation filter is a pre-storedsequence of L bits. When the processor 302 determines that a matchbetween the correlation filter 864 and a consecutive sequence of bitswithin the bit sequence received from the demodulator of the HMD 102 hasoccurred, the processor 302 determines that the L bit sequence is found.

In one embodiment, a time tA of the pulse, e.g., a time of maximumamplitude, a time of peak intensity, etc., of the detected beam hitsignal is modified by the processor 302 based on a time tB ofidentification of the bit sequence window position within the L bitsequence. The L bit sequence is identified by using the correlationfilter 864. For example, the processor 302 determines the time tA as atime at which the peak intensity of the pulse occurs and determines thetime tB at which the processor 302 identifies the bit sequence windowposition within the L bit sequence by applying the correlation filter864. The L bit sequence is received from the processor of the HMD 102 bythe processor 302 via the link 109 (FIG. 1A). The time tA is measured bythe clock source of the HMD 102 and the time tB is measured by a clocksource of the computing device 107. Instead of using the time tA, theprocessor 302 applies the time tB as a correction to the time of hit tothe operations 456 and 458 (FIGS. 4B and 4C) to determine acorresponding orientation of the beam 110 with respect to the origin ofthe xyz co-ordinate system of the projector 104.

FIG. 8F is a diagram of an embodiment of a portion of the Lissajouspattern 500 to illustrate a number of times a match between acorrelation filter and the L bit sequence occurs along the Lissajouspattern 500. For example, a match between a correlation filter and the Lbit sequence occurs a number of times, shown as dots in FIG. 8F, acrossthe Lissajous pattern 500.

FIG. 8G is a diagram of an embodiment of the portion 506 a of theLissajous pattern 500 to illustrate an M-sequence technique in which theL bit sequence is used to determine the location of the photosensor 105a within the diameter d1 of the Lissajous pattern 500. When anintensity, e.g., the maximum amplitude, the peak intensity, the maximumamplitude, etc., of the detected beam hit signal that is used todetermine the orientation of the photosensor 105 a with respect to thexyz co-ordinate system of the projector 104 is detected, the location ofthe photosensor 105 is anywhere along the diameter d1 of the Lissajouspattern 500. For example, the clock source of the HMD 102 measures atime at a center of the pulse of the detected beam hit signal andprovides the time to the processor 302 to determine the location of thephotosensor 105 a at the time. The location of the photosensor 105 aanywhere along the diameter d1 provides an uncertainty interval thatextends+/−a radius of the beam 110 from a trajectory, e.g., theLissajous axis, etc., of the Lissajous pattern 500. In the M-sequencetechnique, when the modulator of the projector 104 modulates a beamgenerated by the beam source of the projector so that the detected beamhit signal is modulated with multiple L bit sequences along the diameterof the Lissajous pattern 500, a number P of the L bit sequence isdemodulated. This value P will be greatest along the center of the beam110 and drop off towards the edge of the beam 110. The uncertaintyinterval becomes limited to two mirrored regions, e.g., region 1 andregion 2, etc., on either side of the trajectory of the Lissajouspattern 500. In the M-sequence technique, the processor 302 narrows thelocation of the photosensor 105 a along the diameter d1 to be at theregion 1 or at the region 2. In an embodiment, some informationregarding divergence of the beam 110 and a range of the beam 110 withinthe space is used.

In one embodiment, with a single beam hit, e.g., when the pulse of thebeam hit signal is generated, etc., multiple locations, e.g., theregions 1 and 2, etc., are obtained across the Lissajous pattern 500 dueto the repetition of the L bit sequence. When there are multiple beamhits, multiple sets of locations across the Lissajous pattern 500 aredetermined by the processor 302. The locations across the Lissajouspattern 500 are narrowed down by the processor 302 by selecting one ormore of the locations that are common across the multiple beam hits. Forexample, when the processor 302 determines that the region 1 for a firstbeam hit is the same as that for a second beam hit, the processor 302determines the location of the photosensor 105 a to be at the region 1along the diameter d1 during the first beam hit instead of being at theregion 2. When enough beam hits to disambiguate the multiple possibleregions are obtained, there is no need to perform synchronization of theclock signal of the projector 104 with the clock signal of the HMD 102.In an embodiment, when a distance occupied by the L bit sequence alongthe trajectory of the Lissajous pattern 500 exceeds a distance betweencrossings of the Lissajous pattern 500 in a lesser dense region, e.g.,the center region 502, etc., of the Lissajous pattern 500, a number ofthe multiple possible regions are reduced by the processor 302.

In an embodiment, an approach is to use a combination of timesynchronization, e.g., synchronization of the clock signal of theprojector 104 with the clock signal of the HMD 102, etc., to somerelatively coarse level and then refine the timing and consequently theorientation of the photosensor 105 a with respect to the xyz co-ordinatesystem of the projector 104 using the M-sequence technique.

In one embodiment, an M-bit sub-sequence of the K number of L-bitsequences is unique both within a disjoint sequence window and alsoacross all disjoint sequence windows, e.g., all disjoint sequencewindows of the K number of L bit sequences, etc. The projectors 104 and702 use a different L bit sequence window. For example, the modulator ofthe projector 104 modulates the beam generated by the beam source of theprojector 104 so that a first sequence window of a first L bit sequencehas a unique combination of bits that identifies the projector 104 andthe modulator of the projector 702 modulates the beam generated by thebeam source of the projector 702 so that a second sequence window of asecond L bit sequence has a unique combination of bits that identifiesthe projector 702. When the demodulator of the HMD 102 provides thesecond L bit sequence to the processor 302, the processor 302 determinesfrom the second sequence window, that the second L bit sequence is sentfrom the projector 702 to identify the projector 702. On the other hand,when the demodulator of the HMD 102 provides the first L bit sequence tothe processor 302, the processor 302 determines from the first sequencewindow, that the first L bit sequence is sent from the projector 104 toidentify the projector 104. An identification of a projector is used bythe processor 302 to determine whether the orientation of thephotosensor 105 a is with respect to an origin of an xyz co-ordinatesystem of the projector or with respect to an origin of an xyzco-ordinate system of another projector.

In an embodiment, there are many parameter choices for a frequency ofthe frame of the

Lissajous pattern 500, a diameter of the beam 110, a spread of the beam110, an optical sweep amplitude of the Lissajous pattern 500, a bit rateof modulation of a beam generated by a beam source, a value of M and Lin the L bit M-sequence, a size of the disc of the photo sensor 105 a.Other parameters include a location of the photo sensor in the Lissajouspattern 500. These parameters affect the measured orientation of thephotosensor 105 a with respect to the xyz co-ordinate system of theprojector 104. An example is provided. Let a Lissajous base frequency,e.g., the frequency f1, be 34,000 hertz, a Lissajous second frequency,e.g., the frequency f2, be 34,060 hertz, a Lissajous repeat period,e.g., a frequency of the frame, be 1/60^(th) second, an optical sweepamplitude be 60 degrees (+/−30 degrees from center), a radius of thedisc of the photosensor 105 a be 1 millimeter, a diameter of the beam110 be 4 millimeters at 3 meters from the projector 104, M be 10 with 2windows of length L, and L be 61 bits. Also, the disc of the photosensor105 a is flat with respect to the projector 104 at the 3 meter distancein a center of a frustum of the projector 104. An approximate time forthe beam 110 to traverse a distance dist across an edge of thephotosensor 105 a of length equal to the photosensor radius iscalculated by the processor 302 as follows. Sweep speed is approximately0.0000147 seconds to sweep 60 degrees. The sweep speed amounts to4,080,000 degrees/second. At d=3 meter range from the projector 104, anangular distance across the distance dist of the photo sensor 105 a is2*arctan(0.5*dist/d)=0.0191 degrees. Consequently, a time taken totraverse the distance dist across the disc of the photosensor 105 a is4.68×10⁻⁹ seconds, which is approximately 5 nanoseconds. To obtain 10bits of data in that approximately 5 nanoseconds, the modulator of theprojector 104 modulates at a bit rate of 2.14 gigabits per second(Gbps). So, an amount of time calculated by the processor 302 for themodulator of the projector 104 to modulate the beam generated by thebeam source of the projector 104 with the 61 bit sequence is 28nanoseconds. Consequently, synchronization between the clock signalgenerated by the projector 104 and the clock signal generated by the HMD102 can have up to a 14 nanosecond error and still unambiguously bedetermined to fall within a L bit windows which allow us to obtain theorientation of the photosensor 105 a with respect to the origin of thexyz co-ordinate system with greater confidence. For example, an accuracyalong the sweep of the beam 110 is better than 1 bit, where 1 bitrepresents approximately 1/10^(th) millimeter.

In an embodiment, there is a global synchronization event, e.g., a radiofrequency (RF) pulse or a global light pulse, etc., that is accurate towithin the sequence of length L, e.g., 61 bits, etc. For example, theprocessor of the projector 104 sends the RF pulse to the projector 702and to the HMD 102 to synchronize the clock sources of the HMD 102 andthe projector 702 (FIG. 7A) with the clock source of the projector 104.To illustrate, at a time the clock signal of the projector 104 pulsesfrom one state to another, e.g., high to low, low to high, etc., theprocessor of the projector 104 sends the RF pulse via the link 117 (FIG.1A) to the HMD 102 and via a link between the projectors 104 and 702 tothe projector 702. Upon receiving the RF pulse, the processor of the HMD102 sends a signal to the clock source of the HMD 102 to synchronize theclock signal of the clock source of the HMD 102 with the clock signal ofthe projector 104. Moreover, the processor of the projector 702 sends asignal to the clock source of the projector 702 to synchronize the clocksignal of the clock source of the projector 702 with the clock signal ofthe projector 104. As another example, a beam generator, e.g., otherthan the beam generator 108, etc., of the projector 104 generates astrobe of a beam and the strobe of the beam is detected by thephotosensor 105 a. At a time the strobe of the beam is detected, theclock signal of the HMD 102 is synchronized with the clock signal of theprojector 104. The global synchronization event occurs once during eachframe or multiple times during each frame.

In an embodiment, the beams 110 and 704 have different light frequenciesand each photosensor 105 a and 105 b has a different filter to detectthe different frequencies. For example, the projector 104 generates thebeam 110 having a first frequency and the projector 702 generates thebeam 704 having a second frequency, which is different from the firstfrequency. The photosensor 105 a is coupled to a first filter and thephotosensor 105 b is connected to a second filter. The first filterfilters in the first frequency but not the second frequency and thesecond filter filters in the second frequency but not the firstfrequency. The photosensor 105 a receives the first frequency from thefirst filter to generate a first detected beam hit signal and thephotosensor 105 b receives the second frequency from the second filterto generate a second detected beam hit signal. Upon determining that thefirst detected beam hit signal has the first frequency, the processor302 determines that the first detected beam hit signal is generated fromthe beam 110 received from the projector 104. Similarly, upondetermining that the second detected beam hit signal has the secondfrequency, the processor 302 determines that the second detected beamhit signal is generated from the beam 704 received from the projector702. A correspondence, e.g., one-to-one relationship, association,mapping, etc., between the first frequency and an identification of theprojector 104 is stored in the memory device 304 and anothercorrespondence between the second frequency and an identification of theprojector 702 is stored in the memory device 304.

FIG. 9 is a diagram of an embodiment of a system 900 to illustrate amethod of synchronization between a device 902 and another device 904.An example of the device 902 is the projector 104 and the device 904 isthe HMD 102. Another example of the device 902 is the projector 702 andthe device 904 is the HMD 102.

In an embodiment, the clock signals of the devices 902 and 904 aresynchronized to a nanosecond level accuracy. For example, a protocolassociated with a computer network's Network Time Protocol (NTP) isused. The device 902 sends a packet X at a local time A₁ and receives ananswer packet Y from the device 904 at a local time A₂. The local timesA₁ and A₂ are measured by the clock source of the device 902. The packetX contains the time A₁. When the device 904 receives the packet X, theclock source of the device 904 timestamps a time B₁ at which the packetX is received. The device 904 then transmits the packet Y at a time B₂and includes in the packet Y, the times B₁ and B₂. The time B₂ ismeasured by the clock source of the device 904. The device 902 receivesthe packet Y at the time A₂. Once the device 902 obtains the packet Y,the processor of the device 902 calculates a round trip network time asa difference between a total time and a processing time taken by thedevice 904. The total time is (A₂−A₁) and the processing time is(B₂−B₁). The round trip network time is considered symmetric and so oneway trip time is T₁=0.5 [(A₂−A₁)−(B₂−B₁)]. So, a true time at the timeA₂ is B₂+T₁. Hence, an offset time T_(off)=B₂+T₁−A₂ is added by theprocessor of the device 902 to the clock signal of the clock source ofthe device 902 to synchronize the clock signal of the device 902 withthe clock signal of the device 904. It should be noted that a packet isgenerated by a processor of a device. For example, the packet X isgenerated by the processor of the device 902 and the packet Y isgenerated by the processor of the device 904. In an embodiment, insteadof the packet X, a request signal is used and instead of the packet Y, aresponse signal is used.

In an embodiment, some statistics are also used to synchronize the clocksignals of the projector 104 and the HMD 102.

In one embodiment, there is some error in the measurement of the timesB₂ and B₁, and therefore in the calculation of B₂−B₁. In the embodiment,there will be some errors in the measurements of the times A₁ and A₂. Inthe embodiment, the packets X and Y are sent several times to measureseveral transmit times and measure several receive times before adecision on the adjustment of the clock signal of the device 902 is madeby the processor of the device 902, or the transmit times and receivetimes are smoothed and filtered. For example, the device 902 sends thepacket X at a local time A₃ and receives the answer packet Y from thedevice 904 at a local time A₄. The local times A₃ and A₄ are measured bythe clock source of the device 902. The packet X contains the time A₃.When the device 904 receives the packet X, the clock source of thedevice 904 timestamps a time B₃ at which the packet X is received. Thedevice 904 then transmits the packet Y at a time B₄ and includes in thepacket Y, the times B₃ and B₄. The time B₄ is measured by the clocksource of the device 904. The device 902 receives the packet Y at thetime A₄. Once the device 902 obtains the packet Y, the processor of thedevice 902 calculates a round trip network time as a difference betweena total time and a processing time taken by the device 904. The totaltime is (A₄−A₃) and the processing time is (B₄−B₃). The round tripnetwork time is considered symmetric and so one way trip time is T₂=0.5[(A₄−A₃)−(B₄−B₃)]. So, a true time at the time A₄ is B₄−T₂. Instead ofan offset time T_(off1)=B₄+T₂−A₄, an average of the offset times T_(off)and T_(off1) is added by the processor of the device 902 to the clocksignal of the clock source of the device 902 to synchronize the clocksignal of the device 902 with the clock signal of the device 904.

In the embodiment in which the L bit sequence and the M-sequence windoware used, the L bit sequence is identified by the processor of the HMD102 or by the processor 302. When f₁=21,060 Hz, f₂=21,000 Hz, and theLissajous pattern 500 has a 30 degree amplitude, the L length of asequence is used to modulate the beam generated by the beam source ofthe projector 104 is 61 bits, M=10, and a data rate of modulating thebeam is 500 megabits per second (Mbps), a time taken by the modulator ofthe projector 104 to modulate the beam is 122 nanoseconds. The M orgreater bit sequence demodulated from a beam hit is unique within theL=61 bit long sequence. The processor of the projector 104 uses theclock signal of the projector 104 to search backwards and forwards the Mbit sequence that is closest among neighboring L bit sequences. So longas an error in synchronization between the clock signal of the HMD 102and the clock signal of the projector 104 is less than 122 nanoseconds,the L bit sequence that is the closest is found. With some tolerances,this reduces a constraint for the synchronization to 100 nanoseconds. Atthis level, in one embodiment, the clock signals of the HMD 102 and theprojector 104 are corrected once per frame using the NTP style out ofband RF communication.

FIG. 10 is a flowchart of an embodiment of a method 1000 forsynchronizing the clock signal of the clock source of the projector 104and the clock source of the HMD 102 by extracting, from the detectedbeam hit signal that is generated by the photosensor 105 a, a frequencyof the modulating signal used to modulate the beam generated by the beamsource of the beam generator 108. The modulating signal is generated insynchronization with the clock signal of the projector. For example, abit 0 of the modulating signal is generated by the modulator of theprojector 104 at a first rising edge of the clock signal of the clocksource of the projector 104 and a bit 1 is generated by the modulator ofthe projector 104 at a second rising edge of the clock signal. The firstrising edge precedes the second rising edge and the bit 0 precedes thebit 1 in a sequence of bits of the modulating signal. As anotherexample, the bit 0 of the modulating signal is generated by themodulator of the projector 104 at a first falling edge of the clocksignal of the clock source of the projector 104 and the bit 1 isgenerated by the modulator of the projector 104 at a second falling edgeof the clock signal. The first falling edge precedes the second fallingedge and the bit 0 precedes the bit 1 in the sequence of bits of themodulating signal.

The demodulator of the HMD 102 is coupled to the photosensor 105 a toreceive the detected beam hit signal that is generated by thephotosensor 105 a by sensing the beam 110. The demodulator of the HMD102 demodulates the detected beam hit signal to generate the modulatingsignal. The modulating signal is provided from the demodulator of theHMD 102 to the processor 302 of the computing device 107 via the link109 (FIG. 1A). The processor 302 determines, in an operation 1002, afrequency of the modulating signal to determine a phase, e.g., timing,etc., of the clock signal generated within the projector 104. The phase,of the clock signal, determined by the processor 302 is provided to aPLL that is coupled to the processor 302. The PLL is located within thecomputing device 107. The PLL compares, in an operation 1004, the phaseof the clock signal generated by the clock source of the projector 104with a phase of the clock signal generated within the HMD 102. The phaseof the clock signal generated within the HMD 102 is received via thelink 109 by the processor 302 from the processor of the HMD 102.

The PLL synchronizes, in an operation 1006, the phase of the clocksignal generated within the HMD 102 with the phase of the clock signalof the projector 104. For example, when the phase of the clock signal ofthe HMD 102 lags or leads the phase of the clock signal of the projector104, the PLL sends a phase difference signal indicating the phasedifference to the processor 302 of the computing device 107, and theprocessor 302 sends the phase difference signal to the clock source ofthe HMD 102 via the link 109 to match the phase of the clock signal ofthe HMD 102 with that of the clock signal of the projector 104. Toillustrate, the phase difference signal is sent from the processor 302to the processor of the HMD 102 via the link 109, and in response toreceiving the phase difference signal, the processor of the HMD 102sends a signal to the clock source of the HMD 102 to match the phase ofthe clock signal generated by the clock source of the HMD 102 with thatof the clock signal generated by the clock source of the projector 104.As another example, when the phase of the clock signal of the HMD 102lags or leads the phase of the clock signal of the projector 104, thePLL sends the phase difference signal to the processor 302 of thecomputing device 107, and the processor 302 sends the phase differencesignal to the clock source of the projector 104 via the link 111 (FIG.1A) to match the phase of the clock signal of the projector 104 withthat of the clock signal of the HMD 102. To illustrate, the phasedifference signal is sent from the processor 302 to the processor of theprojector 104 via the link 111, and in response to receiving the phasedifference signal, the processor of the projector 104 sends a signal tothe clock source of the projector 104 to match the phase of the clocksignal generated by the clock source of the HMD 102 with that of theclock signal generated by the clock source of the projector 104.

The synchronization is performed to improve an accuracy of theorientation, e.g., the beam angles θx, θy, and θz, etc., of the beam 110and of the position of the photosensor 105 a with respect to the originof the xyz co-ordinate system of the projector 104. Both the orientationof the beam 110 and the position of the photosensor 105 a are dependenton the timing at which the detected beam hit signal is generated by thephotosensor 105 a, as described above with reference to FIG. 4B.

In one embodiment, the time of hit at which the beam 110 is detected bythe photosensor 105 a is the same as a time at which the detected beamhit signal is generated by the photosensor 105 a upon detecting the beam110.

In an embodiment, a rise time period for the photosensor 105 a togenerate the beam hit signal is accounted for by the processor 302 indetermining the time of hit. For example, when an initiation time atwhich the photosensor 105 a initiates to detect the beam hit signal isreceived by the processor 302 as the time of hit, the processor 302subtracts the rise time period from the initiation time at which thephotosensor 105 a to modify the time of hit and the modified time of hitis used to determine the orientation of the photosensor 105 a withrespect to the origin of the xyz co-ordinate system of the projector 104instead of the time of hit.

In an embodiment, an accuracy of the orientation of the photosensor 105a with respect to the origin of the xyz co-ordinate system of theprojector 104 is dependent on synchronization of the clock signals ofthe HMD 102 and the projector 104. For example, for obtaining a 1millimeter precision at a 3 meter distance from the projector 104, thebeam 110 moves quickly across the origin of the xyz co-ordinate systemof the projector 104 or about the z axis of the xyz co-ordinate systemof the projector 104. At the 3 meter distance, 1 millimeter represents a0.00033 radian displacement. When a frustum of the projector 104 is 60degrees, f₁=21,060 Hertz, and f₂=21,000 Hertz, movement of the MEMSmirror 112 along the X-axis of the MEMS mirror 112 is represented as x=Asin (2π tf1) and a speed of movement of the MEMS mirror 112 along theX-axis of the MEMS mirror 112 is dx/dt, which is:

$\begin{matrix}{\frac{dx}{dt} = {A\; 2\pi\; f\; 1{\cos\left( {2\pi\; t\; f\; 1} \right)}}} & {{equation}\mspace{14mu}(4)}\end{matrix}$where A is an amplitude of movement of the MEMS mirror 112 along theX-axis of the MEMS mirror 112 to form the Lissajous pattern 500 (FIG.5), and t is time. The speed will be highest at t=0 when the cosine is1.0 and the highest speed is 69,285 radians/second. So to traverse the0.00033 radians, it will take 4.8 nanoseconds for the MEMS mirror 112.When the clock signals of the projector 104 and the HMD 102 are notsynchronized by 4.8 nanoseconds, as much as 1 millimeter of precisionassociated with the orientation of the photosensor 105 a with respect tothe origin of the xyz co-ordinate system is reduced.

In one embodiment, the clock source of the projector 104 is atemperature compensated crystal oscillator (TCXO) and the clock sourceof the HMD 102 is a TCXO. In each of the TCXOs, 1 to 3 parts per partsper million (ppm) drift is achieved. So in case the TCXOs are startedsimultaneously, they will drift by 2 microseconds after one second ofoperation. In an embodiment, the drift is corrected every frame, e.g.,at 16.667 milliseconds, etc. So, the drift at the end of 16.667milliseconds is 33 nanoseconds. In the embodiment, the clock signal ofthe HMD 102 or of the projector 104 is corrected 33 times each frame.

FIG. 11 is a diagram to illustrate averaging of intensities, e.g.,amplitudes, etc., of detected beam signals to determine the orientationof the photosensor 105 a with respect to the origin of the xyzco-ordinate system of the projector 104. At edges, e.g., an edge 1102,etc., of the Lissajous pattern 500, the beam 110 traverses thephotosensor 105 a such that there is no gap in an area on thephotosensor 105 a covered by the beam 110 when the beam 110 turns aroundfrom a first direction to a second direction. The second direction isopposite to the first direction. For example, the second direction is ina −y direction of a y-axis when the first direction is in a +y directionof the y-axis. As another example, the second direction is in a −xdirection of the x-axis when the first direction is in a +x direction ofan x-axis. As another example, the beam 110 turns around as a result ofa turning around the edge 1102 of the Lissajous pattern 500.

The photosensor 105 a of the HMD 102 detects a first intensity when thebeam 110 traverses in the first direction and detects a second intensitywhen the beam 110 traverses in the second direction to generate detectedbeam hit signals, which are provided to the processor of the HMD 102.Moreover, the processor of the HMD 102 receives from the clock source ofthe HMD 102 times at which the first and second intensities aredetected, e.g., a first time at which the first intensity is detectedand a second time at which the second intensity is detected. The timesare sent by the processor of the HMD 102 to the processor 302 of thecomputing device 107 via the link 109 (FIG. 1A). The processor 302averages the times of detection of the first and second beam intensitiesto calculate an average time.

The processor 302 determines a location of the beam 110 in the Lissajouspattern 500 based on the average time, the time of start of theLissajous pattern 500, and the frequencies f1 and f2 of the MEMS mirror112 in a manner similar to that described in the operation 456 (FIG.4B). For example, the processor 302 determines from the time of thestart of the pattern, the frequency f1, the frequency f2, and theaverage time, the location of the beam 110 on the Lissajous pattern 500.To illustrate, the processor 302 of the computing device 107 determinesthat the time of start corresponding to an initial location in theLissajous pattern 500. Moreover, the processor 302 determines that theinitial location changes according to the frequencies f1 and f2 of theMEMS mirror 112 to achieve the location of the beam 110 on the Lissajouspattern 500. The location of the beam 110 corresponds to the averagetime. In one embodiment, a correspondence, e.g., a one-to-one relation,etc., among the frequencies f1 and f2, the average time, the time ofstart, and the location of the beam 110 on the Lissajous pattern 500 isstored in the memory device 304 of the computing device 107. Thecorrespondence is accessed by the processor 302 to determine thelocation of the beam 110 on the Lissajous pattern 500 from the time ofstart, the frequencies f1 and f2, and the average time. The location ofthe beam 110 on the Lissajous pattern 500 provides the orientation,e.g., the beam angles θx, θy, and θz, etc., of the beam 110 with respectto the origin of the xyz co-ordinate system of the projector 104. Byusing the average of the two times, the orientation of the beam 110 withrespect to the origin of the xyz co-ordinate system of the projector 104is more accurate than if a single beam intensity is used at the edge1102 of the Lissajous pattern 500.

FIG. 12 is a flowchart of an embodiment of a method 1200 forsynchronization of the clock signal that is generated within the HMD 102with the clock signal that is generated within the projector 104. Thebeam 110 that follows the pattern on the MEMS mirror 112 is sensed bythe photosensor 105 a multiple times during the frame, e.g., one roundof execution, etc., of the pattern to generate a detected beam hitsignal. The photosensor 105 a sends intensities of the detected beamsignal to the processor 302 of the computing device 107 via theprocessor of the HMD 102 and the link 109 (FIG. 1A). The processor ofthe HMD 102 also receives, in an operation 1202, times at which theintensities are detected from the clock source of the HMD 102 and sendsthe times to the processor 302 via the link 109. The processor 302backtracks, e.g., reverses, etc., in an operation 1204, the times todetermine the time of start of the pattern, e.g., a time at which thepattern starts, a time at which the frame starts, etc. For example, theprocessor 302 determines from the times at which the intensities aredetected and the frequencies f1 and f2, the time of start of thepattern. To illustrate, the processor 302 determines from the times atwhich the intensities are detected, times associated with movement ofthe MEMS mirror 112 at the frequencies f1 and f2, the time of start ofthe pattern. As another illustration, a correspondence, e.g., aone-to-one relationship, an association, etc., between a time at whichone of the intensities is detected, times associated with movement ofthe MEMS mirror 112 at the frequencies f1 and f2, and the time of startof the pattern is stored in the memory device 304 of the computingdevice 107, and the processor 302 accesses the correspondence todetermine the time of start.

In one embodiment, a relative time difference between two of the timesprovides a unique signature that is backtracked by the processor 302 todetermine the time of start of the pattern. As an illustration, acorrespondence, e.g., a one-to-one relationship, an association, etc.,between a time difference between the times at which two consecutiveintensities are detected by the photosensor 105 a, times associated withmovement of the MEMS mirror 112 at the frequencies f1 and f2, and thetime of start of the pattern is stored in the memory device 304 of thecomputing device 107, and the processor 302 accesses the correspondenceto determine the time of start.

The processor 302 determines, in an operation 1206, whether the clocksignal of the projector 104 is synchronized to the time of start that isdetermined in the operation 1204. For example, the processor 302receives a phase of the clock signal of the projector 104 via theprocessor of the projector 104 and the link 111 (FIG. 1A), anddetermines whether the time of start is in phase with the phase of theclock signal. Upon determining that the clock signal of the projector104 is not synchronized to the time of start that is determined in theoperation 1204, the processor 302 sends a control signal to the clocksource of the projector 104 via the processor of the projector 104 andthe link 111 to synchronize the clock signal to the time of start. As anexample, the processor of the projector 104 sends a control signal tothe clock source of the projector 104 so that the clock signal generatedby the clock source is in phase with the time of start. To illustrate,the clock signal generated by the clock source of the projector 104 issynchronized with the time of start when a rising edge or a falling edgeof the clock signal occurs at the same time as the time of start.

In an embodiment, the synchronization of the clock signal to the time ofstart determined in the operation 1204 is performed by a PLL of theprojector 104. For example, the processor of the projector 104 sends acontrol signal having the time of start to the PLL of the projector 104.The PLL of the projector 104 is connected to the processor of theprojector 104 and to the clock source of the projector 104. The PLL ofthe projector 104 compares the time of start with the clock signal ofthe clock source of the projector 104 and synchronizes the time of startwith the clock signal to generate a PLL signal. The PLL signal is sentto the clock source of the projector 104 so that there is no phase lagor phase lead between the time of start and the clock signal of theclock source of the projector 104.

It should be noted that in an embodiment, when a signal, e.g., adetected beam signal, etc., is provided to a processor, the signal isconverted from an analog form to a digital form before being sent to theprocessor. For example, an analog-to-digital converter is connectedbetween the photosensor 105 a and the processor to convert the detectedbeam hit signal from an analog form to a digital form.

In an embodiment, the frame is repeated at a pre-determined repeatinterval, e.g., 60 Hertz (Hz), 50 Hz, 40 Hz, etc.

FIG. 13 is an isometric view of an HMD 1300, which is an example of theHMD 102 (FIG. 1A). The HMD 1300 includes bands 1302 and 1304 that go tothe back of the head of the user when worn by the user 106. Moreover,the HMD 1300 includes earphones 1306A and 1306B, e.g., speakers, etc.,that emanate sound associated with a virtual environment, e.g., a gameenvironment, a virtual tour environment, etc., that is generated byexecution of a computer program, e.g., a game program, a virtualenvironment generation program, etc. The HMD 1300 includes lenses 1308Aand 1308B that allows the user to view a virtual environment that isdisplayed on a display screen of the HMD 1300. A groove 1380 rests on anose of the user 106 to support the HMD 1300 on the nose.

In some embodiments, an HMD 1300 is worn by the user 106 in a mannersimilar to which sunglasses, glasses, or reading glasses are worn by theuser 106.

FIG. 14 illustrates a system for interactive game play of a video game,in accordance with an embodiment described in the present disclosure.The user 106 is shown wearing the HMD 102. The HMD 102 is worn in amanner similar to glasses, goggles, or a helmet, and is configured todisplay a video game or other content to the user 106. The HMD 102provides an immersive experience to the user 106 by virtue of itsprovision of display mechanisms (e.g., optics and display screens) inclose proximity to eyes of the user 106 and the format of content thatis delivered to the HMD 102. In one example, the HMD 102 providesdisplay regions to each eye of the user 106 and the display regionsoccupy large portions or even the entirety of a field-of-view of theuser 106.

In one embodiment, the HMD 102 is connected to the computing device 107.The connection to computing device 107 is wired or wireless. Thecomputing device 107, in one embodiment, is any general or specialpurpose computer, including but not limited to, a game console, apersonal computer, a laptop, a tablet, a mobile device, a smart phone, atablet, a thin client, a set-top box, a media streaming device, a smarttelevision, etc. In some embodiments, the HMD 102 connects directly to acomputer network 1410, e.g., the Internet, an Intranet, a local areanetwork, a wide area network, etc., which allows for cloud gamingwithout the need for a separate local computer. In one embodiment, thecomputing device 107 executes a video game (and other digital content),and output the video and audio from the video game for rendering by theHMD 102. The computing device 107 is also sometimes referred to hereinas a client system, which in one example is a video game console.

The computing device 107, in some embodiments, is a local or remotecomputer, and the computer runs emulation software. In a cloud gamingembodiment, the computing device 107 is remote and is represented by aplurality of computing services that are virtualized in data centers,where game systems/logic is virtualized and distributed to the user 106over the computer network 1410.

The user 106 operates the hand-held controller 115 to provide input fora virtual environment. In one example, the hand-held controller 115captures image of the environment, e.g., a real-world environment, etc.,in which the user 106 is located. These captured images are analyzed todetermine a location and movements of the user 106, the HMD 102, and thecontroller 115. In one embodiment, the controller 115 includes a light(or lights) which are tracked to determine its location and orientation.Additionally, as described in further detail below, in one embodiment,the HMD 102 includes one or more lights, which are tracked as markers todetermine the location and orientation of the HMD 102 in substantialreal-time during a display of a virtual environment.

The hand-held controller 115, in one embodiment, includes one or moremicrophones to capture sound from the real-world environment. Soundcaptured by a microphone array is processed to identify a location of asound source. Sound from an identified location is selectively utilizedor processed to exclusion of other sounds not from the identifiedlocation. Furthermore, in one embodiment, the hand-held controller 115or the HMD 102 includes multiple image capture devices (e.g.stereoscopic pair of cameras), an infrared (IR) camera, a depth camera,and combinations thereof.

In some embodiments, the computing device 107 executes games locally onprocessing hardware of the computing device 107. The games or content isobtained in any form, such as physical media form (e.g., digital discs,tapes, cards, thumb drives, solid state chips or cards, etc.) or by wayof download from the computer network 1410.

In an embodiment, the computing device 107 functions as a client incommunication over the computer network 1410 with a cloud gamingprovider 1412. The cloud gaming provider 1412 maintains and executes thevideo game being played by the user 106. The computing device 107transmits inputs from the HMD 102 and the controller 1404 to the cloudgaming provider 1412, which processes the inputs to affect the gamestate of the video game being executed. The output from the executingvideo game, such as video data, audio data, and haptic feedback data, istransmitted to the computing device 107. The computing device 107further processes the data before transmission or directly transmits thedata to the relevant devices. For example, video and audio streams areprovided to the HMD 102, whereas a vibration feedback command isprovided to the hand-held controller 115.

In one embodiment, the HMD 102 and the hand-held controller 115 arenetworked devices that connect to the computer network 1410 tocommunicate with the cloud gaming provider 1412. For example, thecomputing device 107 is a local network device, such as a router, thatdoes not otherwise perform video game processing, but facilitatespassage of network traffic. The connections to the computer network 1410by the HMD 102 and the hand-held controller 115 are wired or wireless.

In some embodiments, content executed on the HMD 102 or displayable on adisplay device 1414, is obtained from any of content sources 1416.Example content sources can include, for instance, internet websitesthat provide downloadable content and/or streaming content. In someexamples, the content can include any type of multimedia content, suchas movies, games, static/dynamic content, pictures, social mediacontent, social media websites, virtual tour content, cartoon content,etc.

In one embodiment, the user 106 is playing a game on the HMD 102, wheresuch content is immersive three-dimensional (3D) interactive content.The content on the HMD 102, while the user 106 is playing, is shared tothe display device 1414. In one embodiment, the content shared to thedisplay device 1414 allows other users proximate to the user 106 orremote to watch along with game play of the user 106. In still furtherembodiments, another player viewing the game play of the user 106 on thedisplay device 1414 participates interactively with user 106. Forexample, another user viewing the game play on the display device 1414controls characters in the game scene, provides feedback, providessocial interaction, and/or provides comments (via text, via voice, viaactions, via gestures, etc.,) which enables the other user who is notwearing the HMD 102 to socially interact with the user 106.

In one embodiment, instead of or in addition to the hand-held controller115, a position and/or orientation of the HMD 102 with respect to theprojector 104 is determined by using the methods described herein.

FIG. 15 illustrates a head-mounted display (HMD) 1500, in accordancewith an embodiment described in the present disclosure. The HMD 1500 isan example of the HMD 102 (FIG. 1A). As shown, the HMD 1500 includes aplurality of photosensors 1502A-H, J and K (e.g., where 1502K and 1502Jare located toward the rear or backside of the HMD headband). Each ofthese photosensors are configured to have specific shapes and/orpositions. The photosensors 1502A, 1502B, 1502C, and 1502D are arrangedon the front surface, e.g., front face, etc., of the HMD 1500. Thephotosensors 1502E and 1502F are arranged on a side surface of the HMD1500. And the photosensors 1502G and 1502H are arranged at corners ofthe HMD 1500, so as to span the front surface and a side surface of theHMD 1500.

Based on one or more of the beams 110 and 704 detected by one or more ofthe photosensors 1502A-H, J and K, a location of the HMD 1500 in theenvironment is determined. It will further be appreciated that some ofthe photosensors cannot see a beam depending upon the particularorientation of the HMD 1500 relative to one of the projectors 104 and702. In some embodiments, inertial sensors are disposed in the HMD 1500,which provide feedback regarding positioning instead of the photosensors1502A-H, J and K. In some embodiments, the photosensors 1502A-H, J and Kand inertial sensors work together, to enable mixing and selection ofposition/motion data.

The HMD 1500, in one embodiment, additionally includes one or moremicrophones. In the illustrated embodiment, the HMD 1500 includesmicrophones 1504A and 1504B located on the front surface of the HMD1500, and a microphone located on a side surface of the HMD 1500. Byutilizing an array of microphones, sound from each of the microphones isprocessed to determine a location of the sound's source. Thisinformation is utilized in various ways, including exclusion of unwantedsound sources, association of a sound source with a visualidentification, etc.

In an embodiment, the HMD 1500 includes one or more image capturedevices. In the illustrated embodiment, the HMD 1500 is shown to includeimage capturing devices 1506A and 1506B. In an embodiment, by utilizinga stereoscopic pair of image capture devices, 3D images and video of thereal-world environment is captured from the perspective of the HMD 1500.Such video is presented to the user 106 to provide the user 106 with a“video see-through” ability while wearing the HMD 1500. That is, thoughthe user 106 cannot see through the HMD 1500 in a strict sense, thevideo captured by the image capture devices 1506A and 1506B nonethelessprovides a functional equivalent of being able to see the real-worldenvironment external to the HMD 1500 as if looking through the HMD 1500.

Such video, in one embodiment, is augmented with virtual elements toprovide an augmented reality experience, or is combined or blended withvirtual elements in other ways. Though in the illustrated embodiment,two cameras are shown on the front surface of the HMD 1500, it will beappreciated that there may be any number of externally facing cameras ora single camera can be installed on the HMD 1500, and oriented in anydirection. For example, in another embodiment, there may be camerasmounted on the sides of the HMD 1500 to provide additional panoramicimage capture of the environment.

FIG. 16 illustrates one example of game play using a client system 1602that is capable of rendering the video game content to the HMD 1500operated by the user 106. In this illustration, a state of a virtualobject, e.g., game content, etc., provided to the HMD 1500 is in a richinteractive 3D space. As discussed above, a state of a virtual object isdownloaded to the client system 1602 or is executed in one embodiment bya cloud processing system. A cloud gaming service 1632 includes adatabase of users 1604, who are allowed to access particular games 1630,share experiences with other friends, post comments, and manage theiraccount information.

The cloud gaming service 1632 stores game data 1606 for specific users,which may be usable during game play, future game play, sharing to asocial media network, or used for storing trophies, awards, status,ranking, etc. Social data 1608 is managed by the cloud gaming service1632. In one embodiment, the social data 1608 is managed by a separatesocial media network, which is interfaced with cloud gaming service 1632over the computer network 1410. Over the computer network 1410, anynumber of client systems 1610 are connected for access to the contentand interaction with other users.

Continuing with the example of FIG. 16, the 3D interactive scene viewedin the HMD 102 includes game play, such as the characters illustrated inthe 3D view, or another virtual environment. One character, e.g. P1,etc., is controlled by the user 106, who is wearing the HMD 102. Thisexample shows a basketball scene between two players, wherein the user106 is dunking a ball on another character in the 3-D view. The othercharacter can be an AI (artificial intelligence) character of the game,or can be controlled by another player or players (Pn). The user 106,who is wearing the HMD 102, is shown moving about in a space of use,where the HMD 102 moves around based on head movements and bodypositions of the user 106. A camera 1612 is shown positioned over thedisplay device 1414 in the room, however, for use of the HMD 102, thecamera 1612 is placed in any location that can capture images of the HMD102. As such, the user 106 is shown turned at about 90 degrees from thecamera 1612 and the display device 1414, as content rendered in the HMD102 is dependent on the direction that the HMD 102 is positioned, fromthe perspective of the camera 1612. Of course, during use of the HMD102, the user 106 will be moving about, turning his/her head, looking invarious directions, as is needed to take advantage of the dynamicvirtual scenes rendered by the HMD 102.

In one embodiment, instead of or in addition to the camera 1612, aposition and/or orientation of the HMD 102 with respect to the projector104 is determined by using the methods described herein.

FIG. 17 illustrates a user wearing the HMD 102, during use, inaccordance with one embodiment. In this example, it is shown that theHMD 102 is tracked 1702 using image data obtained from captured videoframes by the camera 1612. Additionally, it is shown that the hand-heldcontroller 115 is also tracked 1704 using image data obtained fromcaptured video frames by the camera 1612. Also shown is theconfiguration where the HMD 102 is connected to a computing system 2202via a cable 1710. The computing system 2202 is an example of thecomputing system 107. In one embodiment, the HMD 102 obtains power fromthe same cable or can connect to another cable. In still anotherembodiment, the HMD 102 has a battery that is rechargeable, so as toavoid extra power cords.

With reference to FIG. 18, a diagram is shown illustrating examplecomponents of an HMD 1800, in accordance with an embodiment described inthe present disclosure. The HMD 1800 is an example of the HMD 102 shownin FIG. 1A. When the HMD 1800 excludes any cameras, the HMD 1800 is anexample of the HMD 102 (FIG. 1A). It should be understood that more orless components can be included or excluded from the HMD 1800, dependingon the configuration and functions enabled. The HMD 1800 includes aprocessor 1802 for executing program instructions. A memory device 1804is provided for storage purposes, and in one embodiment, includes bothvolatile and non-volatile memory. A display 1806 is included whichprovides a visual interface that the user views.

The display 1806 is defined by one single display screen, or in the formof a separate display screen for each eye of the user 106. When twodisplay screens are provided, it is possible to provide left-eye andright-eye video content separately. Separate presentation of videocontent to each eye, for example, provides for better immersive controlof 3D content. As described herein, in one embodiment, the second screenis provided with second screen content of the HMD 1800 by using theoutput for one eye, and then formatting the content for display in atwo-dimensional (2D) format. The one eye, in one embodiment, is theleft-eye video feed, but in other embodiments is the right-eye videofeed.

A battery 1808 is provided as a power source for the HMD 1800. In otherembodiments, the power source includes an outlet connection to power. Inother embodiments, an outlet connection to power and the battery 1808are provided. A motion detection module 1810 includes any of variouskinds of motion sensitive hardware, such as a magnetometer 1812, anaccelerometer 1814, and a gyroscope 1816.

An accelerometer 1814 is a device for measuring acceleration and gravityinduced reaction forces. Single and multiple axis (e.g., six-axis)models are able to detect magnitude and direction of the acceleration indifferent directions. The accelerometer 1814 is used to senseinclination, vibration, and shock. In one embodiment, threeaccelerometers are used to provide the direction of gravity, which givesan absolute reference for two angles (world-space pitch and world-spaceroll).

A magnetometer 1812 measures the strength and direction of the magneticfield in the vicinity of the HMD 1800. In one embodiment, threemagnetometers are used within the HMD 1800, ensuring an absolutereference for the world-space yaw angle. In one embodiment, themagnetometer 1812 is designed to span the earth magnetic field, which is±80 microtesla. Magnetometers are affected by metal, and provide a yawmeasurement that is monotonic with actual yaw. The magnetic field iswarped due to metal in the environment, which causes a warp in the yawmeasurement. If necessary, this warp is calibrated using informationfrom other sensors such as the gyroscope or the camera. In oneembodiment, the accelerometer 1814 is used together with magnetometer1812 to obtain the inclination and azimuth of the HMD 1800.

A gyroscope 1816 is a device for measuring or maintaining orientation,based on the principles of angular momentum. In one embodiment, threegyroscopes provide information about movement across the respective axis(x, y and z) based on inertial sensing. The gyroscopes help in detectingfast rotations. However, the gyroscopes drift overtime without theexistence of an absolute reference. To reduce the drift, the gyroscopesare reset periodically, which can be done using other availableinformation, such as positional/orientation determination based onvisual tracking of an object, accelerometer, magnetometer, etc.

A camera 1818 is provided for capturing images and image streams of thereal-world environment. In one embodiment, more than one camera(optionally) is included in the HMD 1800, including a camera that isrear-facing (directed away from the user when the user is viewing thedisplay of the HMD 1800), and a camera that is front-facing (directedtowards the user when the user is viewing the display of the HMD 1800).Additionally, in an embodiment, a depth camera 1820 is included in theHMD 1800 for sensing depth information of objects in the real-worldenvironment.

The HMD 1800 includes speakers 1822 for providing audio output. Also, inone embodiment, a microphone 1824 is included for capturing audio fromthe real-world environment, including sounds from the ambientenvironment, speech made by the user, etc. In an embodiment, the HMD1800 includes tactile feedback module 1826 for providing tactilefeedback to the user. In one embodiment, the tactile feedback module1826 is capable of causing movement and/or vibration of the HMD 1800 soas to provide tactile feedback to the user.

Photosensors 1830 are provided to detect one or more of the beams. Acard reader 1832 is provided to enable the HMD 1800 to read and writeinformation to and from a memory card. A USB interface 1834 is includedas one example of an interface for enabling connection of peripheraldevices, or connection to other devices, such as other portable devices,computers, etc. In various embodiments of the HMD 1800, any of variouskinds of interfaces may be included to enable greater connectivity ofthe HMD 1800.

In an embodiment, a Wi-Fi module 1836 is included for enablingconnection to the computer network via wireless networking technologies.Also, in one embodiment, the HMD 1800 includes a Bluetooth module 1838for enabling wireless connection to other devices. A communications link1840 is included for connection to other devices. In one embodiment, thecommunications link 1840 utilizes infrared transmission for wirelesscommunication. In other embodiments, the communications link 1840utilizes any of various wireless or wired transmission protocols forcommunication with other devices.

Input buttons/sensors 1842 are included to provide an input interfacefor the user. Any of various kinds of input interfaces may be included,such as buttons, gestures, touchpad, joystick, trackball, etc. In oneembodiment, an ultra-sonic communication module 1844 is included in HMD1800 for facilitating communication with other devices via ultra-sonictechnologies.

In an embodiment, bio-sensors 1846 are included to enable detection ofphysiological data from the user. In one embodiment, the bio-sensors1846 include one or more dry electrodes for detecting bio-electricsignals of the user through the user's skin, voice detection, eye retinadetection to identify users/profiles, etc.

The foregoing components of HMD 1800 have been described as merelyexemplary components that may be included in HMD 1800. In variousembodiments described in the present disclosure, the HMD 1800 may or maynot include some of the various aforementioned components. Embodimentsof the HMD 1800 may additionally include other components not presentlydescribed, but known in the art, for purposes of facilitating aspects ofthe present invention as herein described.

In one embodiment, the HMD 1800 includes light emitting diodes, whichare used in addition to the photosensors 1830 to determine a positionand/or orientation of the HMD 1800. For example, the LEDs and a cameralocated within the environment in which the HMD 1800 is located are usedto confirm or deny a position and/or orientation that is determinedusing the photosensors 1830 and the methods described herein.

It will be appreciated by those skilled in the art that in variousembodiments described in the present disclosure, the aforementionedhandheld device is utilized in conjunction with an interactiveapplication displayed on a display to provide various interactivefunctions. The exemplary embodiments described herein are provided byway of example only, and not by way of limitation.

In one embodiment, clients and/or client devices, as referred to herein,may include HMDs, terminals, personal computers, game consoles, tabletcomputers, telephones, set-top boxes, kiosks, wireless devices, digitalpads, stand-alone devices, handheld game playing devices, and/or thelike. Typically, clients are configured to receive encoded videostreams, decode the video streams, and present the resulting video to auser, e.g., a player of a game. The processes of receiving encoded videostreams and/or decoding the video streams typically includes storingindividual video frames in a receive buffer of the client. The videostreams may be presented to the user on a display integral to client oron a separate device such as a monitor or television.

Clients are optionally configured to support more than one game player.For example, a game console may be configured to support two, three,four or more simultaneous players (e.g., P1, P2, . . . Pn). Each ofthese players receives or shares a video stream, or a single videostream may include regions of a frame generated specifically for eachplayer, e.g., generated based on each player's point of view. Any numberof clients are local (e.g., co-located) or are geographically dispersed.The number of clients included in a game system vary widely from one ortwo to thousands, tens of thousands, or more. As used herein, the term“game player” is used to refer to a person that plays a game and theterm “game playing device” is used to refer to a device used to play agame. In some embodiments, the game playing device may refer to aplurality of computing devices that cooperate to deliver a gameexperience to the user.

For example, a game console and an HMD cooperate with a video serversystem to deliver a game viewed through the HMD. In one embodiment, thegame console receives the video stream from the video server system andthe game console forwards the video stream, or updates to the videostream, to the HMD and/or television for rendering.

Still further, an HMD is used for viewing and/or interacting with anytype of content produced or used, such video game content, moviecontent, video clip content, web content, advertisement content, contestcontent, gamboling game content, conference call/meeting content, socialmedia content (e.g., posting, messages, media streams, friend eventsand/or game play), video portions and/or audio content, and content madefor consumption from sources over the internet via browsers andapplications and any type of streaming content. Of course, the foregoinglisting of content is not limiting, as any type of content can berendered so long as it can be viewed in the HMD or rendered to a screenor screen of the HMD.

In one embodiment, clients further include systems for modifyingreceived video. For example, a client performs further rendering, tooverlay one video image on another video image, to crop a video image,and/or the like. As another example, clients receive various types ofvideo frames, such as I-frames, P-frames and B-frames, and to processthese frames into images for display to a user. In some embodiments, amember of clients is configured to perform further rendering, shading,conversion to 3-D, conversion to 2D, distortion removal, sizing, or likeoperations on the video stream. A member of clients is optionallyconfigured to receive more than one audio or video stream.

Input devices of clients includes, for example, a one-hand gamecontroller, a two-hand game controller, a gesture recognition system, agaze recognition system, a voice recognition system, a keyboard, ajoystick, a pointing device, a force feedback device, a motion and/orlocation sensing device, a mouse, a touch screen, a neural interface, acamera, input devices yet to be developed, and/or the like.

A video source includes rendering logic, e.g., hardware, firmware,and/or software stored on a computer readable medium such as storage.This rendering logic is configured to create video frames of the videostream based on the game state. All or part of the rendering logic isoptionally disposed within one or more graphics processing unit (GPU).Rendering logic typically includes processing stages configured fordetermining the three-dimensional spatial relationships between objectsand/or for applying appropriate textures, etc., based on the game stateand viewpoint. The rendering logic produces raw video that is encoded.For example, the raw video is encoded according to an Adobe Flash®standard, HTML-5, .wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv,Lagarith, MPG-x, Xvid, FFmpeg, x264, VP6-8, real video, mp3, or thelike. The encoding process produces a video stream that is optionallypackaged for delivery to a decoder on a device. The video stream ischaracterized by a frame size and a frame rate. Typical frame sizesinclude 800×600, 1280×720 (e.g., 720p), 1024×768, 1080p, although anyother frame sizes may be used. The frame rate is the number of videoframes per second. In one embodiment, a video stream includes differenttypes of video frames. For example, the H.264 standard includes a “P”frame and a “I” frame. I-frames include information to refresh all macroblocks/pixels on a display device, while P-frames include information torefresh a subset thereof. P-frames are typically smaller in data sizethan are I-frames. As used herein the term “frame size” is meant torefer to a number of pixels within a frame. The term “frame data size”is used to refer to a number of bytes required to store the frame.

In some embodiments, the client is a general purpose computer, a specialpurpose computer, a game console, a personal computer, a laptopcomputer, a tablet computer, a mobile computing device, a portablegaming device, a cellular phone, a set-top box, a streaming mediainterface/device, a smart television or networked display, or any othercomputing device capable of being configured to fulfill thefunctionality of a client as defined herein. In one embodiment, a cloudgaming server is configured to detect the type of client device which isbeing utilized by the user, and provide a cloud-gaming experienceappropriate to the user's client device. For example, image settings,audio settings and other types of settings may be optimized for theuser's client device.

FIG. 19 illustrates an embodiment of an Information Service Providerarchitecture. Information Service Providers (ISP) 1902 delivers amultitude of information services to users 1900-1, 1900-2, 1900-3,1900-4, etc., geographically dispersed and connected via the computernetwork 1410. In one embodiment, an ISP delivers one type of service,such as stock price updates, or a variety of services such as broadcastmedia, news, sports, gaming, etc. Additionally, the services offered byeach ISP are dynamic, that is, services can be added or taken away atany point in time. Thus, the ISP providing a particular type of serviceto a particular individual can change over time. For example, a user isserved by an ISP in near proximity to the user while the user is in herhome town, and the user is served by a different ISP when the usertravels to a different city. The home-town ISP will transfer therequired information and data to the new ISP, such that the userinformation “follows” the user to the new city making the data closer tothe user and easier to access. In another embodiment, a master-serverrelationship is established between a master ISP, which manages theinformation for the user, and a server ISP that interfaces directly withthe user under control from the master ISP. In another embodiment, thedata is transferred from one ISP to another ISP as the client movesaround the world to make the ISP in better position to service the userbe the one that delivers these services.

ISP 1902 includes Application Service Provider (ASP) 1906, whichprovides computer-based services to customers over the computer network1410. Software offered using an ASP model is also sometimes calledon-demand software or software as a service (SaaS). A simple form ofproviding access to a particular application program (such as customerrelationship management) is by using a standard protocol such as HTTP.The application software resides on the vendor's system and is accessedby users through a web browser using HTML, by special purpose clientsoftware provided by the vendor, or other remote interface such as athin client.

Services delivered over a wide geographical area often use cloudcomputing. Cloud computing is a style of computing in which dynamicallyscalable and often virtualized resources are provided as a service overthe computer network 1410. Users do not need to be an expert in thetechnology infrastructure in the “cloud” that supports them. In oneembodiment, cloud computing are divided in different services, such asInfrastructure as a Service (IaaS), Platform as a Service (PaaS), andSoftware as a Service (SaaS). Cloud computing services often providecommon business applications online that are accessed from a webbrowser, while the software and data are stored on the servers. The termcloud is used as a metaphor for the Internet (e.g., using servers,storage and logic), based on how the Internet is depicted in computernetwork diagrams and is an abstraction for the complex infrastructure itconceals.

Further, ISP 1902 includes a Game Processing Server (GPS) 1908 which isused by game clients to play single and multiplayer video games. Mostvideo games played over the Internet operate via a connection to a gameserver. Typically, games use a dedicated server application thatcollects data from players and distributes it to other players. This ismore efficient and effective than a peer-to-peer arrangement, but itrequires a separate server to host the server application. In anotherembodiment, the GPS establishes communication between the players andtheir respective game-playing devices exchange information withoutrelying on the centralized GPS.

Dedicated GPSs are servers which run independently of the client. Suchservers are usually run on dedicated hardware located in data centers,providing more bandwidth and dedicated processing power. Dedicatedservers are the preferred method of hosting game servers for mostPC-based multiplayer games. Massively multiplayer online games run ondedicated servers usually hosted by the software company that owns thegame title, allowing them to control and update content.

Broadcast Processing Server (BPS) 1910 distributes audio or videosignals to an audience. Broadcasting to a very narrow range of audienceis sometimes called narrowcasting. The final leg of broadcastdistribution is how the signal gets to the listener or viewer, and itmay come over the air as with a radio station or TV station to anantenna and receiver, or may come through cable TV or cable radio (or“wireless cable”) via the station or directly from a network. TheInternet may also bring either radio or TV to the recipient, especiallywith multicasting allowing the signal and bandwidth to be shared.Historically, broadcasts have been delimited by a geographic region,such as national broadcasts or regional broadcast. However, with theproliferation of fast internet, broadcasts are not defined bygeographies as the content can reach almost any country in the world.

Storage Service Provider (SSP) 1912 provides computer storage space andrelated management services. SSPs also offer periodic backup andarchiving. By offering storage as a service, users can order morestorage as required. Another major advantage is that SSPs include backupservices and users will not lose all their data if their computers' harddrives fail. Further, in an embodiment, a plurality of SSPs have totalor partial copies of the user data, allowing users to access data in anefficient way independently of where the user is located or the devicebeing used to access the data. For example, a user can access personalfiles in the home computer, as well as in a mobile phone while the useris on the move.

Communications Provider 1914 provides connectivity to the users. Onekind of Communications Provider is an Internet Service Provider (ISP)which offers access to the Internet. The ISP connects its customersusing a data transmission technology appropriate for delivering InternetProtocol datagrams, such as dial-up, DSL, cable modem, fiber, wirelessor dedicated high-speed interconnects. The Communications Provider canalso provide messaging services, such as e-mail, instant messaging, andSMS texting. Another type of Communications Provider is the NetworkService provider (NSP) which sells bandwidth or network access byproviding direct backbone access to the Internet. Network serviceproviders, in one embodiment, include telecommunications companies, datacarriers, wireless communications providers, Internet service providers,cable television operators offering high-speed Internet access, etc.

Data Exchange 1904 interconnects the several modules inside ISP 1902 andconnects these modules to users 1900 via the computer network 1410. DataExchange 1904 covers a small area where all the modules of ISP 1902 arein close proximity, or covers a large geographic area when the differentmodules are geographically dispersed. For example, Data Exchange 1988includes a fast Gigabit Ethernet (or faster) within a cabinet of a datacenter, or an intercontinental virtual area network (VLAN).

Users 1900 access the remote services with client device 1920, whichincludes at least a CPU, a display and an input/output (I/O) device. Theclient device can be a personal computer (PC), a mobile phone, anetbook, tablet, gaming system, a personal digital assistant (PDA), etc.In one embodiment, ISP 1902 recognizes the type of device used by theclient and adjusts the communication method employed. In other cases,client devices use a standard communications method, such as html, toaccess ISP 1902.

FIG. 20 is a diagram of an embodiment of an HMD 2000. The HMD 2000 is anexample of the HMD 102 (FIG. 1A). The HMD 2000 includes an array ofphotosensors. Some of the photosensors are labeled as 2006. Not allphotosensors of the HMD 2000 are labeled to avoid cluttering. Thephotosensor 2006 is an example of the photosensor 105 a. Thephotosensors of the HMD 2000 are distributed vertically along a width ofa front face 2002 of the HMD 2000 and also distributed horizontallyacross a height of the front face 2002 of the HMD 2000. Also, in anembodiment, some of the photosensors of the HMD 2000 are located on aheadband 2004 of the HMD 2000. When orientations of two or more of thephotosensors are known with respect to the projector 104, theorientation of the HMD 2000 with respect to the projector 104 isdetermined by the processor 302 of the computing device 107. In oneembodiment, each photo sensor 2006 of the HMD 2000 is identified by theprocessor 302 of the computing device 107 by a unique identificationcode.

Arrows are provided to show directions in which photosensors 2006 arepointing. For example, a lens of the photosensor 2006 at a top leftcorner of the HMD 2000 is facing up but is angled with respect to avertical 2007. As another example, a lens of the photosensor 2006 in amiddle left corner of the HMD 2000 is facing at an angle of 90 degreeswith respect to the vertical 2007. As yet another example, a lens of thephotosensor 2006 in a middle top corner of the HMD 2000 is facingvertically up to form an angle of zero degrees with respect to thevertical 2007. As still another example, a lens of the photosensor 2006in a middle bottom corner of the HMD 2000 is facing vertically down toform an angle of 180 degrees with respect to the vertical 2007. As stillanother example, a lens of the photosensor 2006 in a middle portion ofthe HMD 2000 is facing out of the paper of FIG. 20 to form an angle of90 degrees with respect to the vertical 2007.

In some embodiments, examples of a pattern in which the photosensors2006 are placed on the surface of the front face 2002 include a randomnonplanar pattern, a zig-zag pattern, a curved pattern, a straightpattern, a combination of a curved and a straight pattern, etc. Invarious embodiments, the photosensors 2006 are placed at differentangles with respect to the front face 2002 and/or the vertical 2007 thanthat illustrated in FIG. 20.

In one embodiment, the HMD 2000 has a front cover 2010, e.g., a glasscover, a plastic cover, a shaded cover, a tinted cover, etc., and thefront cover has embedded lenses 2008 that go over the front face 2002.Each lens 2008 is located on top of a corresponding photosensor 2006 ofthe HMD 2000. The lens 2008 directs, e.g., refracts, etc., stray lightfrom the beam 110 to facilitate incidence of the beam 110 on thecorresponding photosensor 2006 that is located below the lens.

In an embodiment, one of the photosensors form a different angle withrespect to the front face front face 2002 of the HMD 2000 compared toanother one of the photosensors.

In one embodiment, all of the photosensors of the HMD 2000 have the sameangle with respect to the front face, e.g., the front surface, etc., ofthe HMD 2000.

In an embodiment, each photosensor 2006 has a curvature. In oneembodiment, the curvature is exposed to light or not.

Embodiments described in the present disclosure may be practiced withvarious computer system configurations including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers and the like. Theembodiments described in the present disclosure can also be practiced indistributed computing environments where tasks are performed by remoteprocessing devices that are linked through a wire-based or wirelessnetwork.

With the above embodiments in mind, it should be understood that theembodiments described in the present disclosure can employ variouscomputer-implemented operations involving data stored in computersystems. These operations are those requiring physical manipulation ofphysical quantities. Any of the operations described herein that formpart of the embodiments described in the present disclosure are usefulmachine operations. Some embodiments described in the present disclosurealso relate to a device or an apparatus for performing these operations.The apparatus can be specially constructed for the required purpose, orthe apparatus can be a general-purpose computer selectively activated orconfigured by a computer program stored in the computer. In particular,various general-purpose machines can be used with computer programswritten in accordance with the teachings herein, or it may be moreconvenient to construct a more specialized apparatus to perform therequired operations.

Some embodiments described in the present disclosure can also beembodied as computer readable code on a computer readable medium. Thecomputer readable medium is any data storage device that can store data,which can be thereafter be read by a computer system. Examples of thecomputer readable medium include a hard drive, a NAS, a ROM, a RAM, aCD-ROM, a CD-R, a CD-RW, a magnetic tape, an optical data storagedevice, a non-optical data storage device, etc. The computer readablemedium can include computer readable tangible medium distributed over anetwork-coupled computer system so that the computer readable code isstored and executed in a distributed fashion.

It should be noted that in some embodiments, any of the embodimentsdescribed herein are combined with any of the remaining embodiments.

Moreover, although some of the above-described embodiments are describedwith respect to a gaming environment, in some embodiments, instead of agame, other environments, e.g., a video conferencing environment, etc.,is used.

Although the method operations were described in a specific order, itshould be understood that other housekeeping operations may be performedin between operations, or operations may be adjusted so that they occurat slightly different times, or may be distributed in a system whichallows the occurrence of the processing operations at various intervalsassociated with the processing, as long as the processing of the overlayoperations are performed in the desired way.

Although the foregoing embodiments described in the present disclosurehave been described in some detail for purposes of clarity ofunderstanding, it will be apparent that certain changes andmodifications can be practiced within the scope of the appended claims.Accordingly, the present embodiments are to be considered asillustrative and not restrictive, and the embodiments are not to belimited to the details given herein, but may be modified within thescope and equivalents of the appended claims.

The invention claimed is:
 1. A method for determining a position of aphotosensor with respect to multiple projectors, comprising: generating,by a beam generator of a projector, a beam; modifying a direction oftravel of the beam using a micro-electro-mechanical systems (MEMS)mirror that moves in a pattern; detecting, by a photosensor, the beam;calculating a time of hit at which the beam is detected; determiningbased on the pattern and the time of hit an orientation of thephotosensor with respect to the projector; generating, by a beamgenerator of an additional projector, an additional beam; modifying adirection of travel of the additional beam using an additional MEMSmirror that moves in an additional pattern; calculating an additionaltime at which the additional beam is detected; determining based on theadditional pattern and the additional time an orientation of theadditional beam; and determining a position of the photosensor withrespect to the projector based on the orientation of the beam and theorientation of the additional beam.
 2. The method of claim 1, whereinthe pattern of the MEMS mirror is a Lissajous pattern or a raster scanpattern, wherein the pattern of the additional MEMS mirror is theLissajous pattern or the raster scan pattern.
 3. The method of claim 1,wherein the beam is a laser beam and the additional beam is a laserbeam.
 4. The method of claim 1, further comprising: modulating the beamwith a first bit pattern; and modulating the additional beam with asecond bit pattern.
 5. The method of claim 4, further comprising:determining, based on the first bit pattern, that the beam is generatedby the projector; and determining, based on the second bit pattern, thatthe additional beam is generated by the additional projector.
 6. Themethod of claim 1, further comprising synchronizing a clock signalgenerated by a controller that detects the beam with a clock signalgenerated by the projector.
 7. The method of claim 1, wherein the beamand the additional beam are time division multiplexed with respect toeach other.
 8. A system determining a position of a photosensor withrespect to multiple projectors, comprising: a projector including a beamgenerator configured to generate a beam, wherein the projector includesa micro-electro-mechanical systems (MEMS) mirror configured to move in apattern to modify a direction of travel of the beam; a controller is incommunication with the projector, wherein the controller has aphotosensor, wherein the photosensor is configured to detect the beam,wherein the controller is configured to calculate a time of hit at whichthe beam is detected; and a computing device in communication with theprojector and the controller, wherein the computing device is configuredto determine based on the pattern and the time of hit an orientation ofthe photosensor with respect to the projector; an additional projectorin communication with the computing device, wherein the additionalprojector includes a beam generator configured to generate an additionalbeam, wherein the additional projector includes an additional MEMSmirror configured to move in the pattern to modify a direction of travelof the additional beam, wherein the controller is configured tocalculate an additional time at which the additional beam is detected,wherein the computing device is configured to determine based on theadditional pattern and the additional time an orientation of theadditional beam; and wherein the computing device is configured todetermine a position of the photosensor with respect to the projectorbased on the orientation of the beam and the orientation of theadditional beam.
 9. The system of claim 8, wherein the pattern of theMEMS mirror is a Lissajous pattern or a raster scan pattern, wherein thepattern of the additional MEMS mirror is the Lissajous pattern or theraster scan pattern.
 10. The system of claim 8, wherein the beam is alaser beam and the additional beam is a laser beam.
 11. The system ofclaim 8, wherein the projector is configured to modulate the beam with afirst bit pattern, and wherein the projector is configured to modulatethe additional beam with a second bit pattern.
 12. The system of claim11: wherein the computing device is configured to determine, based onthe first bit pattern, that the beam is generated by the projector; andwherein the computing device is configured to determine, based on thesecond bit pattern, that the additional beam is generated by theadditional projector.
 13. The system of claim 8, wherein the computingdevice is configured to synchronize a clock signal generated by thecontroller that detects the beam with a clock signal generated by theprojector.
 14. The method of claim 8, wherein the beam and theadditional beam are time division multiplexed with respect to eachother.
 15. A system comprising: a controller having a photosensorconfigured to detect a beam, wherein the controller is configured tocalculate a time of hit at which the beam is detected; and a computingdevice in communication with the controller, wherein the computingdevice is configured to determine based on a pattern of motion of amicro-electro-mechanical systems (MEMS) mirror and the time of hit anorientation of the beam, wherein the controller is configured to detectan additional beam to generate an additional beam hit signal, whereinthe controller is configured to calculate an additional time of hit atwhich the additional beam is detected, wherein the computing device isconfigured to determine based on a pattern of motion of an additionalmicro-electro-mechanical systems (MEMS) mirror and the additional timeof hit an additional orientation of the additional beam, wherein thecomputing device is configured to determine a position of thephotosensor based on the orientation of the beam and the orientation ofthe additional beam.
 16. The system of claim 15, wherein the pattern ofthe MEMS mirror is a Lissajous pattern or a raster scan pattern, whereinthe pattern of the additional MEMS mirror is the Lissajous pattern orthe raster scan pattern.
 17. The system of claim 15, wherein the beam isa laser beam and the additional beam is a laser beam.
 18. The system ofclaim 15, wherein the beam is encoded with a first bit pattern, whereinthe additional beam is encoded with a second bit pattern.
 19. The systemof claim 15, wherein the beam and the additional beam are time divisionmultiplexed with respect to each other.
 20. The system of claim 15,wherein the controller is a head-mounted display.